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APPENDIX  I 


ACQUISITION  QUESTIONNAIRE 
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AI-2 


ACQUISITION  OVERVIEW 


question  l:  What  are  the  major  problems  encountered  in  system 
acquisitions  you  are  familiar  with?  (Respondents 
where  asked  to  check  those  that  apply.) 


answers:  %  of  Responses  Problem  Areas 


100 

Time  slippage 

91 

Cost  overruns 

100 

Failure  of  Service  to  fully  specify 
what  is  requi red  1 

73 

Failure  of  Contractor(s)  to  meet 
specifications 2 

27 

Failure  of  Service  and  Contractor(s) 
to  deal  realistically  with  new  technologies 

27 

Justification  of  budget  commitments 

82 

Competition  for  limited  funds1 

100 

Changing  requirements1  2 

91 

User/developer  interface1  2 

64 

Implementation  does  not  evolve  from 
requirements1  2 

64 

Reinventing  the  "wheel"1 

18 

Determination  of  how  and  when  multiple 
contractor  assignments  should  begin  and  end 

64 

Quality  of  management 

55 

Rapid  management  turnover  rate 

82 

Lack  of  incentives  to  call  a  halt  to  any 
step  of  acquisition2 

64 

Clarification  of  knowns  from  unkowns 1 

64 

No  specific  early  warning  signals  for 
error  detection  available  as  management 
tools1  2 

'Importance  in  providing  PDSS  Planning  and  Implementation. 

"Most  probable  result  of  developer  and  user  having  a  poor  communication 
interface . 


AI-3 


at 


QUESTION  2: 


ANSWERS : 


Can  you  cite  an  example(s)  of  a  lesson  learned  from  a 
previous  success  or  failure  that  was  or  is  being  used 
in  the  acquisition  of  a  new  systems? 


(Bullets  indicate  individual  responses) 

Bureaucratic  process  interferred  with  learning  process; 
competition  (e.g.,  can  not  go  to  guys  th3t  can  do  a 
good  job);  too  much  guidance  from  Top;  every  program  seems 
to  be  different. 


System  must  be  fully  tested  in  contractor's  plant 
(laboratory)  before  being  allowed  to  go  into  field  tests. 
Systems  integration  testing  has  become  recognized  as  a 
key  factor  before  going  into  the  field.  Find-fix-test  or 
"debug"  in  the  field  is  costly  and  should  be  minimized, 
[author's  note:  see  below  for  alternative  opinion] 

Capable,  competent  people  must  be  dedicated  to  at  the 
rate  of  one  (1)  person/$250K  per  year.  They  must  visit 
the  contractor  site,  create  tests,  review  progress  and 
interact  with  the  developer  on  at  least  a  weekly  basis. 

We  have  had  fewer  problems  doing  it  this  way. 


We  never  learn! 


•  Equal  emphasis  on  software  development  as  on  hardware- 
lesson  learned  from  TACFIRE, 

§  Test  planning  must  parallel  system  development-- 
lesson  learned  from  TACFIRE.  Used  in  Battery  Computer 
System.  Problem  clarifying  known-knowns,  unknown- 
kncws,  known-unknowns,  and  unknown-unknowns. 


«  Lesson  learned:  Tactical  Display  System. 

Underestimating  schedule/cost  factors  in  development 
programs  (e.g.,  TACFIRE/TOS/PLRS). 


Several  with  TOS.  Based  on  lessons  learned  (primarily 
from  TACFIRE)  we  are:  (1)  insuring  properly  sequenced 
deliverables  with  appropriate  intermediate  deliverables; 
(2)  software  contract  properly  structured;  (3)  provisions 
made  for  necessary  visibility  throughout  development;  and 
(4)  overall  program  schedule  modified  away  from  "all 
success". 
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ACQUISITION  REVIEW:  Question  2  answers  continued 


9  One  major  lesson--continuity  of  management,  and  direction 
causing  restart  and  redo.  The  Design-to-Price  EW  systems, 
aware  of  this  problem  experienced  relative  success  by 
maintaining  long  term,  high  level  management  throughout  the 
system  design,  development,  T&E  and  acquisition. 


•  Success:  Functional  logistics  analysts  worked  in  teams 
with  ADP  analysts.  Provided  direct  user/developer  co¬ 
ordination;  implemented  established  requirements;  caused 
invalid  requirements  to  be  identified  and  withdrawn. 

9  Success:  Find-Fix-Test  mode  provided  rapid  identification 
of  problems,  relative  rapid  resonse  in  fixing;  and  gave  user 
much  higher  level  of  confidence  in  the  system. 

•  Failure:  No  identification  of  Post  Deployment  SW  Support. 

Continues  to  allow  systems  to  be  fielded  and  PDSS  handled  as 
expensive  afterthought:  incorrect  documentation  identified  in  CDRL 1 
no  language  guides;  no  upfront  money  to  allow  support  personnel 

to  learn  with  development  of  system. 

9  In  general  we  seem  to  continue  in  our  practice  of  the  same 
old  stupidities.  Perhaps  one  small  area  of  light  is  appear¬ 
ing  in  the  specification  process  where  we  are  beginning  to 
specify  software  requirements  as  a  separate  entity  from  and 
on  equal  footing  with  hardware. 


•  The  PATRIOT  system  is  being  developed  and  too  late  into  the 
development  cycle  it  was  "Oh,  by  the  way  we  did  not  get  any 
guidance  or  requirements."  Now  the  developer  is  trying  to 
play  catch-up.  There  is  a  serious  shortage  of  personnel, 
funds,  facilities  and  management. 


ASSUMPTIONS 


There  are  certain  assumptions  we  all  make  as  system  managers  or  as  system 
engineers.  Respondents  were  asked  to  check  the  items  below  that  they 
agreed  with.  They  were  also  asked  to  list  others  that  are  not  included 
in  the  items  listed  below. 
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ASSUMPTIONS :  Question  1  percentage  of  responses  continued . 


of  Responses 

Assumptions 

64 

Individual  systems'  needs  are  not  in  general  unique1  : 

91 

There  are  some  very  basic  properties  that  are  generic 
to  all  systems 

45 

Advanced  technology  more  often  evolves  within  an 
environment  where  systems  are  large  and  complex  3  4 

64 

Various  disciplines  are  recognizing  the  importance  of 
solving  the  problems  of  communication  5 

100 

There  is  now  an  emphasis  on  "methodologies"  6 

-  hierarchical  decomposition 

-  front-end  emphasis 

-  integration  of  "modules"  throughout  a 
development  process 

-  requirements  and  specification  languages 

82 

The  power  of  a  definition  is  not  fully  realized7 

100 

The  power  of  simplification  is  not  fully  realized  8 

73 

The  power  of  commonality  and  abstraction  is  not  fully 
realized  5 

55 

The  power  of  flexibility  and  extendability  is  not 
ful ly  real ized  5  9  1 0 

82 

The  difference  between  "good"  modularity  and  "bad" 
modularity  still  needs  to  be  better  understood  5  11  12 

64 

Algorithms  and  techniques  are  often  misused  inter¬ 
changeably 

91 

Sometimes  the  same  problems  exist  in  the  development5 
of  methodologies  as  exist  in  the  problems  the  method¬ 
ologies  are  intended  to  address 

Management  approach  is  general;  technical  properties  are  unique. 

2Systems  is  rather  broad.  Prefer  categorization  of  tactical  systems 
since  there  are  some  basic  differences  between  business  batch 
processing  versus  tactical -real  time  systems. 

3Apollo,  yes;  laser,  no;  general  rule:  reasonable  technology 
evolves  with  money. 

4Due  to  the  large  funds  available  to  PM. 

5Very  important 

6Not  yet  within  Army  as  a  whole. 

7The  lack  of  concise  definitions  cost  dearly! 

8Change  "realized"  to  "explored". 

9Humans  are  terrible  at  expressing  requirements  in  advance. 

10Change  "power"  to  "necessity". 

'Modularity  itself  needs  better  definition. 

I2Change  "understood"  to  "explained". 
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ASSUMPTIONS :  Question  1  percentage  of  responses  continued. 


%  of  Responses  Assumptions 

82  Often  in  the  attempt  to  compare  methodologies  there  is 

the  risk  of  comparing  apples  and  oranges 

-  techniques  addressing  very  different  problems 

-  techniques  intending  to  address  the  problem, 
but  not  effectively  addressing  it  or  not 
addressing  it  at  all 

-  techniques  against  no  requirements  or  require¬ 
ments  not  well  defined 

-  the  "syntax"  of  methodologies  instead  of  the 
"semantics"  of  methodologies 

-  techniques  that  are  rid  of  preconceived  notions13 
with  preconceived  notions 

-  techniques  addressing  the  wrong  problem 

-  techniques  with  respect  to  completion  or  amount13 
of  use  rather  than  with  respect  to  the  problems 
they  are  solving 


91  The  behavior  of  large  systems  and  their  environment  is  bei 

observed  without  the  advantage  of  formal  definition  tools 

55  Sometimes  one  set  of  methods  is  replaced  by  a  new  set  of 

methods  with  only  the  solutions  to  problems  of  the  old 
method  as  a  consideration  5  13  14 

73  More  work  needs  to  be  done  in  the  area  of  knowing  how  to 

integrate  methods  and  then  integrating  a  system  or  systems 

54  The  choice  of  methodologies  used  can  make  the  difference13 

between  understanding  a  problem  and  not  understanding  a 
problem 

54  The  system  problems  we  are  attempting  to  solve  are  very  is 

basic  ones,  such  as 

-  how  do  people  learn 

-  how  do  people  think 

-  how  do  people  communicate 

-  how  do  people  resource  allocate 


l3Statement  unclear. 

14For  example,  we  never  incorporate  new  technology  to  do  same  function, 
but  we  always  add  requirements. 

15At  a  certain  level  of  abstraction 

16Add  common  sense  to  list  and  add  "why"  for  every  "how". 

17For  example,  a  good  manaqer  has  to  be  a  leader  but  reverse  is  not  true. 
18These  areas  need  much  emphasis. 
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ASSUMPTIONS :  Question  1  percentage  of  responses  continued 


%  of  Responses  Assumptions 

64  The  system  problems  we  are  attempting  to 

solve  are  not  unlike  those  in  "older 
fields.5  **  20 

45  The  system  problems  we  are  attempting  to 

solvp  provide  much  more  visibility  to 
some  basic  issues  than  related  but  more 
developed  fields.1  3 

36  A  whole  new  set  of  basic  principles  is 

needed  for  developing  systems.21  22 

2i  More  often  than  not,  the  user  is  the  one 

who  makes  the  decision  as  to  what  is 
technology  and  as  to  what  is  not.  23 

answers:  Others  (specify) 

•  Communication  through  a  formal  structure  is  almost 
non-existent  across  the  multi-disciplines  engaged 
in  the  development  process. 

•  Linking  of  requirements  to  specifications  to  sub¬ 
systems  to  modules  to  operational  criteria  is  not 
a  clear  hierarchy  of  steps. 

t  Architecture  needs  tools  for  requirement  experi¬ 
mentation  as  well  as  the  engineer  needs  tools. 


19Qualify  as  "some"  system  problems, 

2  “Change  "we  are  attempting  "  to  "we  should  be  attempting". 

21No,  there  are  some  meaningful  principles  needed  to  complement 
those  currently  existing. 

2 2Change  "basic"  to  "organized". 

2 3Not  in  the  Army. 
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UNDERSTANDING  THE  PROBLEM 


Question  1 


ANSWERS 


1  Very 


» 

i 


What  types  of  data  do  you  gather  to  understand  better  the 
environment  of  a  particular  user?  (Respondents  were  asked 
to  check  those  that  applied.) 


%  of  Respones 


Types  of  Data 


64 

64 

64 

73 

82 


Greatest  complaints 
Wish  lists 

Desirable  properties  already  in  the 
user's  system 

Manual  orocesses 

Interfaces1 

answers:  Others  (specify) 


•  Operational  tests  (tests  human  factors  reliability) 

•  Failures 

•  Definition  and  assignment  of  the  threat 

•  System  inputs/outputs  (i.e.,  what  the  real  function 
to  be  performed  as  opposed  to  the  method  currently 
trying  to  carry  out  the  function). 

•  Requirements  inherent  in  his  mission 

•  User  involvement  in  design  decisions 

•  To  the  maximum  extent  possible  I  would  ignore  the 
how  he  does  it  today  or  thinks  he  wants  it  done. 
Include  existing  desirable  properties  where  not 
deletereous  (user  acceptability), 

•  Operational  doctrine. 

•  Chain  of  command  -  protocol. 

•  Response  time  requirements 

•  Implementation  of  manual  processes  into  automated 
systems  merely  speeds  up  a  perhaps  inefficient 
operation.  The  system  concept  need  be  understood. 
The  products  desired  stated.  Then  review  the 
desired  result  against  list  above  and  validate  or 
refute  as  appropriate. 


important. 
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QUESTION  2: 


ANSINERS : 


QUESTION  3: 


ANSWERS: 


What  types  of  user  support  do  you  envision  as  desirable 
for  your  own  environment?  (Respondents  asked  to  check 
those  that  applied. ) 


%  of  Responses 

36  Model  of  the  user  remains  fixed  and 

user  performs  same  functions1 

-  help  speed-up  functions  with  more 
standardization  or  automated  tools 
which  support  his  non-existent  or 
already  existing  support  aids 

-  provide  additonal  information  at 
certain  checkpoints  which  help  him 
make  better  decisions  faster 

-  prevent  him  from  making  errors,  or 
recover  from  errors  already  made 

-  provide  best  decision  making  inform¬ 
ation  at  the  latest  time  possible 
(dynamic  as  opposed  to  static) 

36  Model  of  user  remains  fixed  but  some  of 

user's  functions  are  replaced  by  automated 
aids 

27  Chanqe  the  model  of  the  user 

55  All  of  the  above 


If  you  could  list  five  (5)  of  the  most  important  lessons  you 
have  learned  in  developing  systems,  what  would  they  be? 


(Bullets  indicate  individual  answers.) 

a)  Spent  time  formally  defining  the  system  requirements  using 
scenarios 

b)  Develop  testinq  scenarios  for  the  requirements  and  final 
system 

c)  Simulate/simulate/simulate 

d)  Get  a  completed  tested  design,  build  it,  then  test 
against  b 

e)  Give  yourself  plenty  of  time 


1  First  model  entire  system. 


UNDERSTANDING  THE  PROBLEM:  Question  3  answers  continued . 


•  a)  Pay  now  or  pay  more  later.  Skipping  formalized  incre¬ 

mental  testing  until  "integration"  is  a  disaster. 

b)  Define  specifically  and  completely  what  the  functional 
performance  envelope  should  be. 

c)  Break-up  functions  into  manageable  groupings  with 
inputs,  outputs,  processing  and  "testable"  testing. 

d)  START-STOP,  G0-N0G0  between  phases,  change  or  be 
able  to  change  vendors.  Include  incentives. 

e)  Continuous  A  and  B  level  configuration  management 
baselines  (with  ECP  changes)  are  mandatory. 

•  a)  Do  your  homework  in  advance -- plan,  estimate,  define 

requirements--most  important,  user  must  be  intimately 
involved  as  system  is  being  built  throughout  develop¬ 
ment  process  for  his  ownership  and  acceptability. 

•  b)  Do  not  try  to  buy  it  cheap.  "Cheap  is  cheap"  -  fully 

fund  in  advance  in  terms  of  time  and  money  or  do  not 
start. 

c)  Build  in  flexibility  and  G0-N0G0  decision  points  and 
use  them  (implies  good  management  and  visibility). 

We  need  incremental  checkpoints.  To  build  a  system  and  then  go 
through  DT-OT  is  too  late.  Then,  only  option  left  is  to  kill  the 
whole  thing.  We  wind  up  buying  it  anyway. 

d)  Sometime  or  other  have  to  stop  changing  requirements  and  freeze 
baseline. 

e)  Treat  software  like  hardware. 


•  a)  Do  a  comprehensive  analysis  of  the  requirements. 

b)  Put  sub-requirements  in  priority  order  so  as  to 
address  more  important  ones. 

c)  Spend  effort  in  front  deciding  which  requirements 
you  can  and  cannot  address  because  of  technology 
or  money. 

d)  Resist  changing  requirements  -  make  sure  user  knows 
effects. 

•  a)  Verbal  and  written  communication  of  requirements  in 

the  language  of  the  writer(s)  is  insufficient  for 
disciplined  action. 

b)  Requirements  vary  over  lead  time  contributing  to 
longer  lead  time  and  life  cycle. 

c)  Schedule  and  dollars  tend  to  be  unrealistic  as  pro¬ 
jected  early  in  the  development  cycle. 
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UNDERSTANDING  THE  PROBLEM:  Question  3  answers  continued . 


d)  Design  reviews  appear  to  suppress  rather  than  expose 
development  problems. 

e)  Development  programs  are  not  sufficiently  flexible 
to  take  advantage  of  ensuing  technology  advances. 

f)  The  final  product  cannot  be  sufficiently  examined  for 
its  acceptability  or  unacceptability;  therefore  it 

is  accepted. 


•  a)  Unambiguous  statement  of  requirements  is  most  important 

consideration. 

b)  Freeze  the  requirements  early  -  postpone  changes  for  later 
implementation. 

c)  Plan  for  problems  -  don't  expect  all  success. 

d)  Need  a  qood  work  breakdown  structure  to  (a)  grasp  total 
problem,  (b)  obtain  visibility  for  control. 

e)  A  few  really  good  people  are  worth  far  more  than  dozens 
of  "bodies."  Corollary:  The  solution  to  major  problems 
is  not  to  throw  more  people  in  to  work  on  them. 

•  a)  Long  term  exposure  to  existing  system  -  user  actions  - 

interfaces. 

b)  Again,  long  term  system  concept  formulation,  with  user 
interaction,  with  final  specification. 

c)  Solidify  specifications  and  stick  to  it  during  the 
the  shortest  phased  development. 

d)  Demonstrate  early  and  often  at  low  level  with  simulations 
of  expensive  modules.  After  successful  demonstration  hide 
the  system.'! 

e)  Program  at  the  beginning  of  the  project  10%  contingency 
costs.  Identify  it  as  such,  any  program  CUTS  below  the 
10%  level  should  cause  reexamination  of  programrproba- 
bility  of  success  is  now  deminished! 

•  a)  People  do  not  understand  ''system." 

b)  Automating  manual  procedures  may  not  develp  an  efficient 
system. 

c)  Failure  to  consider  operational  life  and  support  require¬ 
ments  after  development. 

d)  Supertalent  depart  for  new  developments  rather  than 
remain  for  support  and  maintenance. 

e)  Requirements  must  state  "WHAT"  and  not  "HOW". 
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UNDERSTANDING  THE  PROBLEM:  Question  3  answers  continued . 


9  a)  The  specification  is  inspecific. 

b)  The  user  has  no  clear  idea  of  his  requirements *, 
problem  rarely  understood,  far  less  clearly 
articulated. 

c)  The  translation  of  requirements  into  a  system 
specification  often  transcends  reality. 

d)  Specification  detail  when  provided  is  often 
irrelevant. 

•  a)  Clearly  define  requirements. 

b)  Identify  resources. 

c)  Watch  the  contract.  We  (Army),  in  general,  are  poor 
contract  writers/managers. 

d)  Reinventing  the  wheel. 

e)  Buying  the  same  items  numerous  times. 


question  4:  What  are  the  kinds  of  problems  have  you  had  in  developing 
systems?  (Respondents  asked  to  check  those  that  applied.) 


answers ;  %  of  Responses 


Problems 


Problem  not  understood 
Complex  systems 

Requirements  always  changing:  new 
ideas  or  errors1 

Not  knowing  how  much  detail  is  needed 
to  understand  a  system  definition2 

Unrealistic  estimates  of  computer  time, 
manpower,  calendar  time,  on-board  computer 
space  and  time 


:Not  controlled. 

2What  constitutes  satisfactory  B-level  specifications? 
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UNDERSTANDING  THE  PROBLEM3 :  Question  4  percentages  of  responses  continued. 

%  of  Responses 

Problems 

91 

Poor  visibility  and  traceability4 

36 

Asynchronous  aspects  of  sotware  and  its  interfaces 

27 

Class  of  problems  include  those  which  are  time- 
critical  and  self-correcting  (feedback) 

73 

Uncertainty  of  how  much  to  test:  Redundancies  and 
omissions 

55 

Standards  and  disciplines  not  defined5 

91 

Ambiguous,  implicit,  too  detailed,  or  incorrect 
requirements 

73 

Fragmentation  of  personnel 

18 

Unknown  hardware  effects  on  software e 

64 

Software  must  accommodate  hardware 

73 

Management  problems  inherent  in  large  systems:  too 
little  or  too  much7 

91 

Difficulty  in  measuring  correctness  of  software8  9 

64 

Software  not  transferable” 

82 

Symptoms  rather  than  root  problems  treated 

73 

System  not  understood 

55 

No  integrated  goals11 

64 

No  integrated  methodology 

3This  list  could  go  on  and  on  . . . 
‘‘And  monitoring. 

5 Not  used,  even  when  defined. 
64hen  in  parallel. 

7Too  structured. 

8And  extent. 

’Very  important. 
l0But  overcoming  this. 

11  On  paper  -  not  in  reality. 
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UNDERSTANDING  THE  PROBLEM:  Question  4  percentages  of  responses  continued . 


%  of  Responses 


Problems 


55 


91 

82 

55 

55 

82 

91 

91 

55 

73 

36 

27 

45 

73 

73 

55 

55 


Structure  of  system  development  process 
not  flexible  enough  to  encourage  multiple 
technologies,  vendors,  competitive  inno¬ 
vation  and  multiple  sourcing12  13  14 

Unrealistic  schedules9 

Limited  resources9 

Difficulty  in  measuring  effectiveness  of 
software  methodology/tools 

Costly  and  lengthy  efforts 

Lack  of  sufficient  documentation:  too 
little  or  to  much15  16 

The  problems  of  Parkinson’s  law 

Poor  communciation 

Communication  lags 

Communication  interfaces  not  defined 

Lurking  errors 

"Man-rated"17 

Cannot  be  verified  in  the  real  world18 

System  does  not  live  up  to  expectations 

A  need  for  automatic  error  detection  schemes 

Lack  of  flexible  reconfiguration  schemes 
during  development  and  real  time 

Misunderstandings  about  capabilities  of 
support  systems 


“Separate  this  list. 

“Government  Dolicy  precludes. 

“Dollar  limited,  not  structure. 

“Vague,  need  intent  of  completeness. 

“Too  much. 

“Unclear  statement  [Author's  note:  This  means  a  system  in  which 
an  error  could  cause  the  death  of  a  human.] 

“Not  biggest  thing  in  Army. 


UNDERSTANDING  THE  PROBLEM :  Question  4  percentages  of  responses  continued . 


r  Responses 

Probl em 

73 

Design  by  auditorium9  19  20 

45 

Paradox  of  redundancy  management  schemes 

82 

Improper  structuring  of  incentives  for 
contractors  and  Government  management 
personnel 9 

73 

Specific  and  narrow  scoped  interests21 

45 

Difficulties  in  personnel  attitudes  toward 
cooperation9  22 

82 

False  economies9  23 

55 

Over  sophistication 

55 

Creation  of  "urgent"  problems  by  failure  to 
anticipate  troubles  or  respond  expeditiously 

ANSWERS : 

Others  (specify) 

•  Reliance  on  contractor  to  tell  Government  what  is 
needed  when  he's  got  it  and  how  great  it  is 

•  Improper  testing-test  to  specifications  vs. 
error  testing 

•  Lack  of  understanding  of  what  software  is 

•  Lack  of  flexibility  for  technology  insertion  of 
minimum  risk  to  schedule  and  cost.  Sunk  cost 
often  precludes  significant  performance  enhance¬ 
ments  . 

•  Design  reviews  that  do  not  reveal  design  problems 

•  lack  of  qualified  project  management  personnel 

•  "Money-rated" 

•  Parkinson  was  an  optimist 

•  We  still  try  to  justify  systems  in  terms  of 
numbers  of  personnel  saved  or  eliminated 


^Specification  by  committee 
20 Requirements  by  committee 
aLack  of  system  perspective 
aNot  invented-here  syndrome'. 

^Regulations  prevent  Army  from  buying  enough-other  side  of  Parkinson's  law. 


questions  5:  Pick  a  system  or  a  target  for  development.  What  types  of 
requirements  affect  its  development1?  (Respondents  were 
asked  to  check  those  that  applied  and  to  add  other  items 
in  areas  where  applicable2  \) 


%  of  Responses 

S.ystem/Tarqet 

45 

Customer" 

73 

Mission 

82 

User 

55 

General  tool  requirements 

64 

Methodology 

64 

Host  facility5 

82 

Support  systems6 

-  within  target  system 

-  within  development  tools  needed/ 
desired  to  develop  system 

-  requirements  imposed  by  tools 
already  developed 

55 

Pre-design  system  execution  requirements 

73 

Standards 

82 

Operational  reliability 

100 

Testing® 

45 

Statistical  gathering® 

1  Seems  to  be  a  misnomer  in  some  instances. 

Customer,  mission  and  user  requirements  inseparable. 

3User  =  Customer  in  Army;  mission  defined  by  user. 

4 PM  is  customer  in  Army  [author's  note:  see  Note  3  for  alternate  opinion]. 
sLeast  important. 

6Add  "within  host  system". 

7Unclear  statement.  Does  this  mean  simulation  or  modelling? 

8Repeatable  and  recursive. 

9This  is  generated  for  testing. 
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UNDERSTANDING  THE  PROBLEM:  Question  5  percentages  of  resDonses  continued . 


%  of  Responses  Problem 

55  Overall  goals10  11 

answers:  Other  (specify) 

•  Funding  (fiscal) 

•  Lack  of  central  responsible  activity 

•  networking  requirement  because  of  its 
potential  complexity  and  influence  of 
design 

•  Management  structure  (i.e.,  the  DoD 
acquisition  system 

question  6:  What  do  you  consider  the  root  problems  to  be  in  developing 
a  system?  (Respondents  asked  to  check  those  that  applied.) 

answers :  %  of  Responses  Root  Problems 

91  Understanding  a  system  and  its  environment12 

64  Communication  within  and  between  systems14 

55  Resource  allocation  within  and  between 

systems 

64  Flexibility,  both  in  development,  and  in 

real  time 

Other  (specify) 

t  Not  understanding  the  problem  to  be  solved  plus 
people  doing  what  they  want  to  do  and  not  what 
they  ought  to  do  equals  disaster 

0  Algorithm  development 

0  System  concept  (i.e.,  central  i zed/decentral  ized) 

0  Scaling  of  subsystems  in  accordance  with  resource 
allocation 

0  Total  life  cycle 

0  Management  ignorance  of  the  difference  between 
hardware  and  software 

“Performance  bounds. 

“Goals  and  objectives,  as  opposed  to  requirements  are  the  first  thinqs  dis¬ 
carded  when  the  going  gets  tough:  they  are  front-end  window  dressing. 

12 Very  important. 

“Plus  system  elements. 

14Not  as  important  as  others 
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CHOOSING  AN  EFFECTIVE  METHODOLOGY 


question  l;  Without  taking  the  advantage  of  fprmal  definition  tools, 
we  are  not  taking  advantage  of  the  power  of  several  side 
benefits.  (Respondents  were  asked  to  check  those  that 
applied. ) 


answers:  %  of  Responses 


Formal  Definition  Tools 


82 

91 

82 

45 

73 

73 

64 


ANSWERS : 


Definition1 
Simplification 
Common a 1 ity 
Abstraction 
Flexibility 
Extendabil ity 

Understanding  what  the  system  is 
that  is  being  defined2 

Others  (specify) 


Readability  by  reviewers  and  users  representing 
various  disciplines. 

•  Trackability  -  can  one  follow  functional  processes 
across  OS-applications  programs  -  common  processes. 

•  Standards  (documentation,  conventions,  data  element 
dictionaries,  protocol) 


question  2:  Which  viewpoints  of  a  system  are  you  most  concerned  with? 

(ResDondents  were  asked  to  check  those  that  applied.) 


answers :  %  of  Response s 


Viewpoints 


45 

g 

82 

64 

82 

45 


Object 

Name 

Definition 

DescriDtion 

Implementation3 

Execution 

Other  Verification  and  Validation 


Please  define. 

2  Appears  redundant  to  1  and  2. 

3What  is  the  difference  between  implementation  and  execution? 

[Author's  note:  ImDlementation  prepares  for  execution,  see  Section  4.0 
of  Volume  1 ,] 
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question  3;  Listed  below  are  a  series  of  questions  and  answers. 

(Respondents  were  asked  to  check  those  answers  that  they 
agreed  with. ) 


ANSWERS:  »  of  Responses 


Questions  and  Answers 

Question,-  How  can  we  tell  if  a  methodology 
will  work  better  than  no  method¬ 
ology  at  all? 

Answer,-  Compare  the  properties  of  the 
methodology  with  those  used  in 
an  existing  development  with 
respect  to  a  well  defined  set  of 
requirements  for  consistency  and 
completeness. 

Question :  How  do  we  choose  between  one  method¬ 
ology  and  another  methodology? 

Answer.-  Compare  the  properties  of  the  two 
methodologies  with  respect  to  a 
well  defined  set  of  requirements 
for  consistency  and  completeness. 

Question;  What  is  the  difference  between  using 
a  methodology  and  using  "smart" 
people? 


Answer:  The  smartest  person,  by  definition, 
would  apply  an  effective  methodology. 

An  effective  methodology  would  far 
exceed  the  advantages  of  a  smart  per¬ 
son  applying  his  techniques  in  an  ad  t 
hoc  manner,  since  all  the  intricacies 
of  a  complex  system  are  by  its  nature 
beyond  the  grasp  of  one  human  being. 

The  designs  of  all  smart  people  must 
be  integrated1*  5  ^ 

55  Question;  How  do  we  use  a  methodology  without 

impacting  deliverables  of  an  on-going 
project. 

Answer.-  Choose  those  aspects  of  the  methodol¬ 
ogy  which  find  errors  or  which  exped-  • 
ite  the  design  and  implementation 
process6 . 


** W i t h  those  of  dumb  people. 

5But  please  fina  me  a  good  smart  person  rather  than  a  methodology 
because  of  the  added  flexibil ity.  But  since  smartes  are  not  available... 

6Might  agree  if  the  answer  was  couched  in  a  different  language.  This  is 
a  short-sighted  approach. 
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CHOOSING  AN  EFFECTIVE  METHODOLOGY;  Question  3  percentages  of  responses  continued . 


%  of  Responses 


Questions  and  Answers 


73 


100 


Question.-  How  do  we  convince  management, 
designers,  and  users  to  use 
different  approaches? 

Answer.-  A  different  methodology  should  be 
demonstrated  within  the  environ¬ 
ment  of  the  people  who  will  use 
it78. 

Question:  What  creativity  is  left  for  the 
engineers  if  a  methodology  has 
constraints? 


Answer;  An  effective  methodology  should 
support  creative  designs  and  not 
constrain  them  from  producing 
better  designs  but  rather  constrain 
them  from  producing  errors9. 


question  4:  What  methodologies  have  you  heard  of  before?  (Respondents 
were  asked  to  check  those  that  applied.) 

answers:  *  of  Responses  Methodologies 


36 

HDM 

73 

SREM 

55 

PSL/PSA 

100 

Structured  Design 

9 

Warnier 

100 

H  IPO 10 

36 

Jackson 

36 

SADT 

55 

Software  Factory 

82 

H0S 

7Success  stories  heard  from  peers  is  most  effective.  It  is  getting 
the  initial  implementation  that  is  hard. 

8And  it  must  be  effective. 

’Smart  assistant! 

I0Not  really  a  methodology,  just  a  tool. 
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CHOOSING  AN  EFFECTIVE  METHODOLOGY:  Question  4  percentages  of  responses  cont 


QUESTION  5 


QUESTION  6: 


%  of  Responses  Methodologies 

Answers .•  Others  (Specify) 

9  t  CARA 

18  •  IQRL 

9  •  SVD 

18  •  Threads 

9  •  Chief  Programmer  Teams 

9  •  SAM 

9  •  SCALD 


;  Which  of  these  are  you  most  familiar  with? 


%>  of  Responses 
45 
45 
36 
9 
9 

27 

18 

9 

9 

9 


Methodologies 
Structured  Design 
HOS 
HIPO 
IORL 

SVD/Threads 

SREM 

PSL/PSA 

HDM 

SCALD 

About  the  same  of  each 


Would  you  recommend  any  of  these  methodologies? 


%  of  Responses 
9 

18 

9 

9 

9 

9 

9 


Methodologies 
Structured  Design 
HOS 
HIPO 
IORL 
SVD 
SREM 
PSL/PSA 
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CHOOSING  AN  EFFECTIVE  METHODOLOGY:  Question  6  percentages  of  responses  cont. 


9  Knowledge  not  deep  enough  to  comment 

18  Most  of  them  over  none  at  all 


question  7-.  What  properties  do  you  consider  to  be  desirable  ones  for  a 

system  development  methodology?  (Respondents  were  asked  to  check 
those  that  applied.) 


ANSWERS: 


of  Responses 


Desirable  Properties 


91 

73 

82 

91 

64 

73 

91 

55 


Techniques  for  defining  systems  which  are 
consistent  and  logically  complete. 

Techniques  which  are  within  themselves 
consistent  and  logically  complete,  both 
with  respect  to  each  other  and  to  the 
system  to  which  they  are  being  applied11. 


A  standard  set  of  definitions  which  reside 
in  a  well  publicized  and  evolving  glossary12. 

Mechanisms  to  define  all  of  the  relation¬ 
ships  which  exist  in  a  system  environment13. 

Mechanisms  to  define  aJQ_  of  the  relation¬ 
ships  which  exist  between  possible  view¬ 
points  (or  development  layers)  of  a  system14. 


Mechanisms  to  consistently  and  completely 
define  an  object  and  its  relationships 


forma 11' 


Provisions  for  modulari tv  of  the  right 
kind  and  prevention  of  separation  of  the 
wrong  kind. 


Provisions  for  a  set  of  primitive  standard 
mechanisms  which  are  used  both  for  defining 
and  verifying  a  system  in  the  form  of  a 
hierarchy. 


nDo  you  mean  "applicability?"  [Author's  note:  yes.] 
teThat  is,  a  well  defined  methodology. 

13Appl  icabil  ity 

14Change  "all"  to  "of  the  relevant" 
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CHOOSING  AN  EFFECTIVE  METHODOLOGY :  Question  7  percentages  of  responses  cont . 


%  of  Responses  Desirable  Properties 

73  Provision  for  an  evolving  set  of  more 

powerful  (with  respect  to  simplicity  and 
abstraction)  mechanisms  based  on  the  standard 
set  of  primitive  mechanisms.15 

TOO  Allow  system  engineers  to  communicate  in  a 

language,  with  common  semantic  primitives 
and  a  famil iar  dialect,  which  is  extensible, 
flexible,  and  serves  as  a  "library"  of 
common  data  and  structure  mechanisms. 

82  Provisions  for  a  development  model ,  including 

a  set  of  definitions,  tools,  and  techniques, 
which  effectively  suDport  a  given  system 
development  process  16  17 

64  Not  only  must  a  methodology  be  effective, 

but  it  must  also  be  able  to  be  used  as  well, 
and  the  results  of  that  use  should  be  made 
available  to  others16  1.e 

64  There  should  be  an  explicit  set  of  rules  that 

must  be  followed  in  order  to  proceed  from  one 
level,  or  one  layer,  to  another  so  that 

-  one  is  able  to  determine  if  a 
function  has  been  properly  decom¬ 
posed  with  respect  to  complete  re¬ 
placement;  no  more,  no  less 

-  one  is  able  to  specify  a  system  with¬ 
out  data  conflicts  or  timing  conflicts 

45  Each  node  on  a  given  hierarchy  integrates 

all  aspects  of  control,  architecture,  and 
viewpoints. 

82  On  a  given  hierarchy  one  knows  when  a  system 

definition  is  complete16. 


lsNot  necessary  for  a  given  system,  but  desirable. 

“Motherhood. 

17Need  flexibility  to  tailor  it. 

“Here  is  where  we  lose  effectiveness  in  the  application  of  technically 
sophisticated  or  not-so  sophisticated  methodologies.  We  must  be  will¬ 
ing  to  apply  people  to  do  the  job  and  often  we  are  unwilling  to  do  so 
in  the  process  of  development. 
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CHOOSING  AN  EFFECTIVE  METHODOLOGY :  Question  7  percentages  of  responses  cont 


%  of  Responses  Desirable  Properties 

Answers.-  Others  (specify) 

•  The  techniques  should  be  easy  to  understand 
and  use  by  the  system  designer. 

•  Technique  should  be  interpreted  by  cognizant 
but  not  super-sophisticated  personnel. 

•  As  a  language,  technique  should  be  of  the 
order  of  current  HOLs. 

•  Methodology  should  affort  traceability  or 
linking  to  lower  level  specs/ programming 
languages . 


REQUIREMENTS  FOR  FORMAL  METHODS 

question  i:  What  kinds  of  ambiguity  have  you  had  experience  with  in 

your  system  development  processes?  (Respondents  were  asked 
to  check  those  that  applied.) 

answers:  %  of  Responses  Kinds  of  Ambiguity 

64  Definition  of  terms 

82  Definition  of  specifications/ 

requirements 

Answers:  Other  (specify) 

9  •  Hierarchy  of  functional  processes  or 

priorities  in  processing. 

9  •  Lack  of  testability  of  components  and 

functions. 


question  2:  Which  terms  have  you  had  difficulty  with?  (Dashes  indicate 
individual  responses.) 

answers :  -  S  o  f  twa  re 

-  Modular 

-  Real-time 

-  Standardization 

-  Interoperability 

-  Embedded 

-  Protocol 


AI-25 


REQUIREMENTS  FOR  FORMAL  METHODS:  Question  2  answers  continued, 


-  Priority 

-  Security 

-  Continuity  of  operations 

-  Specifications 

-  What  the  system  is  to  do  (lack  of 
specifically  of  functional  allocations) 

-  Requirements 

-  A,  B,  C  Level  Specs 

-  Design 

-  Test 

-  Evaluation 

-  Qua! ity  Assurance 

-  Production  Engineering 

-  System  Definition 

-  Reliability,  Availability,  Maintainability  (RAM) 

-  Verification  and  Validation1 


question  3:  What  types  of  units  should  be  defined  in  a  system 

definition?  (Respondents  were  asked  to  check  those  that 
appl ied. ) 


answers :  %  of  Responses 


Types  of  Units 


91 

91 

82 


9 


Data  types 
Functions 

Structures  (relationships  between 
functions) 

Answers:  Others  (specify) 

•  limiting  constraints 


’Definiton  of  one  respondent:  Verification  is  levels  and  layers, 
validation  is  execution,  that  is,  does  what  it  is  supposed  to  do. 
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question  6:  Do  you  check  for  redundant  defintions?  How? 


answers:  %  of  Responses 

45 
18 
9 


Response 

Yes 

No 

Not  really 


how:  %  of  Responses  Method 

Eyeball 

Manually2 

Multiple  names  for  common  items 
across  modules/functions 


question  7:  Do  you  use  any  standards  or  methods  to  encourage 
modularity3? 


ANSWERS:  S_Sf_Re§Eonsgi  -Response 

36  Yes 

27  No 

Do  these  methods  always  help,  if  no,  are  some 

types  of  modularity  methods  error  prone? 

answers:  %  of  Responses  Response 

9  Yes 

9  No 

10  Sometimes 

answer:  Nine  percent  of  the  respondents  felt  that  some 

types  of  modularity  methods  are  error  prone. 

91%  did  not  respond 

2And  this  is  time  consuming. 

3Aren't  you  really  talking  about  composition  methods?  [Author's  note:  yes] 
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REQUIREMENTS  FOR  FORMAL  METHODS:  Question  7  answers  continued , 


If  yes,  how  do  these  methods  help?  (Bullets  Indicate 
individual  answers.) 

•  Surface  inconsistencies  and  redundancies, 

•  Allowing  some  level  of  reconfiguration  to 
satisfy  changing  requirements  -  assists  in 
phased  development 

•  Need  management  controls  to  limit  exceeding 
modularity  constraints 


SYSTEM  VIEWPOINTS 

question  i:  What  definition  techniques  dp  you  use?  (Respondents  were 
asked  to  check  those  that  applied,) 


answers :  %  of  Responses 


Definition  Techniques 


64 

9 


Hierarchical  decomposition 

Algebraic  data  types 

Answer.'  Other  (explain) 

a  Only  the  process  afforded  by  MIL-STD  490- 
A,B,C  Level  Specs 


question  2:  If  you  use  a  hierarchical  decomposition  technique,  which 
one  do  you  use? 


answers:  (Bullets  indicate  individual  answers.) 

•  HIP0 

•  HDM 

t  Top  Down  Structured  Analysis/Design 

•  Informal  Decomposition 
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question  3:  What  description  techniques  do  you  use  for  system  require¬ 
ments  or  specifications1?  (Respondents  were  asked  to 
check  those  that  applied  and  to  indicate  which  ones  applied.) 


answers:  %  of  Responses 
18 

9 

45 

73 

36 

9 

9 


Description  Techniques 

Specification/Requirement 

Language 

Program  Design  Language 
Higher  Order  Languages 

English 

Graphics  (diagrams) 

Others 

None 


Which  ones? 

PSL/PSA 

PDL 

CMS- 2,  SPL-1,  ADA  (1981), 
TACPOL,  IFTRAN,  PASCAL, 
FORTRAN,  COBOL 

Digital  System  Diagram 
MIL-STD  490 


question  4:  What  implementation  techniques  do  you  use?  (Respondents 

were  asked  to  check  those  that  applied  and  which  ones  applied.) 


answers.-  %  of  Responses  Implementation  Techniques  Which  ones? 


55 

Compiler  driven 

TACPOL,  FORTRAN, 
HAL/S 

18 

Analyzer 

DAVE/PET 

36 

Manual 

9 

Others 

MIL-STD  490 

question  5:  What  execution 
asked  to  check 

techniaues  do  you  use? 
those  that  appl led. ) 

(Respondents  were 

answers:  %  of  Responses 

Execution 

Techniques 

55 

Simulation 

18 

Hybrid 

36 

Static  automatic 

‘Army  does  not  dictate;  one  exception  is  ATLAS. 
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SYSTEM  VIEWPOINTS:  Question  5  percentages  of  responses  continued . 


%  of  Responses 


Execution  Techniques 


27  Static  manual 

36  Dynamic  automatic 

18  Dynamic  manual 

9  Modelling 

27  Interface 

45  Performance 

9  Off-nominal  Stress  Tests 


question  6:  When  do  you  think  it  will  be  possible  to  go  directly  from 
a  requirement/specification  to  execution  code?  (Respon¬ 
dents  were  asked  to  check  projected  time  frames.; 


answers:  %  of  Responses 

18 

9 

0 

18 

18 


Time  Frames 

One  (1 )  year2 
Five  (5)  years 
Ten  (10)  years 
Over  ten  (10)  years 
Never3 


THE  SYSTEM  DEVELOPMENT  PROCESS 

question  l:  What  components  do  you  think  should  go  into  a  model45? 

(Respondents  were  asked  to  check  those  that  applied.) 

answers :  %  of  Responses  Components 

55  Goals 

64  Methodology 


Possible  in  one  year,  practical  in  five,  wide-spread  use  in  ten. 
3You  must  design  the  system. 

4Unclear  statement:  Also  do  you  mean  system  model,  development  process 
model?  [Author's  note:  System  development  process.) 

sThis  is  more  difficult  than  6  above  to  obtain. 


AI-31 


THE  SYSTEM  DEVELOPMENT  PROCESS:  Question  1  percentages  of  responses  cont. 


r  Responses 

Components 

55 

Definitions 

45 

Library 

55 

Disciplines  and  their  checklists 
(Management,  design6,  implementati 
verification,  documentation) 

55 

Phases  and  their  checklists 

55 

Relationships  betv/een  and  within 
phases  and  disciplines 

45 

Tools  and  techniques 

36 

Personnel  structure 

Answers.- 

Other  (Specifiy) 

•  Standard  operating  procedures 

•  Restrictions 

•  System  inputs/functional  characteristics, 
outputs,  system  interfaces  and  communi¬ 
cation  paths 

•  Testing 


question  2:  We  make  the  assumption  that  the  disciplines  listed  below 
correspond  directly  to  the  system  viewpoints  listed  beside 
them.  Do  you  agree?6  8 


Disci  pi ines 

Management 

Documentation 

Design 

Resource  Allocation 

Verification7 

Others 


System  Viewpoints 

Control 
Description 
Definition 
Impl ementation 
Execution7 


6In  general  yes,  but  they  overlap. 

7A1 1  except  this  one. 

8e.g.,  resource  allocation  ambiguous.  Part  of  the  management  process 
one  hand,  may  pertain  to  internal  system  resources  also. 
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SYSTEM  DEVELOPMENT  PROCESS:  The  answer  to  Question  2. 


answers :  %  of  Responses 

27 

9 

64 


Response 

Yes 

No 

No  response 


•  Would  prefer: 

Discipline 
resource  allocation 
execution 

verification 


Viewpoint 


segmentation 

assignment  or 
modularization 

implementation  satisfies 
requirement 


question  3:  We  believe  that  an  ultimate  goal  for  a  system  development 
manager  is  to  replace  himself  by  automation.  Do  you  agree? 
(Author  asked  for  explanations.) 


answers.-  %  of  Responses 

Response 

18 

Yes 

55 

No 

27 

No  response 

Explanations:  (Bullets  indicate  individual  responses.) 

t  Yes,  tired  of  working. 

t  No,  it  is  to  work  himself  out  of  a  job. 

•  Yes,  for  requirements,  provided  tools  are 
available,  the  resultant  system  should  do 
precisely  everything  that  was  intended. 

•  No,  there  are  elements  of  management  which 
are  human  and  not  quantifiable 

•  No,  have  to  insure  that  requirements  are 
really  being  met. 
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SYSTEM  DEVELOPMENT  PROCESS:  Question  3  explanations  continued . 


QUESTION  4: 


ANSWERS: 


•  Mot  really.  The  system  development  managers 
must  set  the  stage  for  the  development  effort, 
determine  the  means  for  measuring  progress, 
and  resolve  the  human  failings  and  misunder^ 
standings  as  they  arise.  The  machine  cannot 
carry  out  these  functions  as  well. 

•  No,  this  is  like  saying  that  a  command  and  con¬ 
trol  system  should  issue  commands  and  control 
their  execution,  That  is  what  a  control  system 
does  on  a  micro  level.  At  the  macro  level,  the 
commander  uses  a  C&C  system  to  assist  him  in  his 
making  decisions,  and  then  assists  him  in  seeing 
that  these  command  decisions  are  communicated 

and  implemented.  The  manager  should  never  be  out  of 
the  loop  either. 

[Author's  comment:  See  Section  4,  Volume  1,  for 
classification  of  leader  vs,  manager,] 

Can  you  envision  a  way  in  which  more  formal,  modular,  or 

communicable  approaches  could  help  you  as  a  manager? 

(Bullets  indicate  individual  responses.) 


%  of  Responses 

Response 

55 

Yes 

45 

No  response 

•  Eliminate  layers  of  management 

•  If  requirements  are  set  forth  completely,  then 
implementation,  testing,  acceptance  and  follow- 
on  modification  and  PD  can  be  controlled  all  with 
minimum  amount  of  resources. 

«  The  most  common  means  of  conveying  ideas,  instruc¬ 
tions,  relationships,  and  meanings  is  the  English 
language  and  it  is  largely  insufficient  in  spoken 
or  written  form  for  system  development  programs 
involving  people  schooled  in  a  variety  of  disciplines. 

•  Yes,  bookkeeping.  Checking  of  design  decisions,  main¬ 
taining  and  checking  descriptions  for  completeness  and 
consistency. 
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SYSTEM  DEVELOPMENT  PROCESS:  Question  4  explanations  continued , 


•  Provide  a  more  accurate  picture  of  indicators 
which  will  allow  more  rapid  management  action. 
This  allows  better  use  of  resources  and  aids 
in  "on  time,  within  cost", 

•  Yes.  (Well,  you  did  not  ask  for  a  description 
of  "how".) 


question  5:  What  are  the  overall  goals  in  developing  your  system? 

(Respondents  were  asked  to  number  in  order  of  importance.) 


answers:  Nine  percent  of  the  participants  did  not  respond. 


OVERAL  GOALS 

Satisfy  the  customer 
Satisfy  yourself,  technically 
Prepare  for  follow-on  work 
Other  (explain) 

•  Help  my  company  make  profit 

•  Make  a  contribution8  3  6 

•  Satisfy  the  bureaucracy9 

•  Satisfy  yourself, 
professionally 


RESPONSES 


1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

3 

3 

2 

4 

2 

2 

1 

2 

1 

2 

2 

3 

3 

3 

2 

4 

2 

.... 

3 

1 

3 

8That  is,  improve  the  technology,  company,  Army,  Country,  etc, 

It  is  possible  to  please  the  customer,  satisfy  yourself,  and, 
yet,  contribute  nothing,  because  the  task  itself  was  not  worth¬ 
while. 

9This  requires  some  constraints  on  satisfaction  of  the  customer. 
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question  6:  Below  is  an  example  of  one  way  to  produce  a  system  design. 

Does  this  method  conform  in  concept  to  your  own?  (Re¬ 
spondents  were  asked  for  comments.) 

Jot  -  jot  down  notes  about  functions  that  are  in  the 
target  system;  organize  and  recorganize  the  notes 
until  they  exist  in  a  hierarchical  form  to  work 
with. 

plan  -  complete  as  much  as  possible  a  commented  hier¬ 
archy  with  questions  which  reflect  specification 
properties  of  target  system  functions.  Use  object 
lists,  library,  standards  as  aid  in  asking  and 
answering  questions. 

interview  -  use  the  definition  to  interview  system, 
support  system  engineers  and  customer  to  fill  in 
the  missing  parts.  Enter  your  own  original 
information  acquired  from  questions  on 
standard  forms  into  the  requirements  data  base. 

Produce  system  -  define  new  specification  mechanisms 
and  incorporate  into  system  definition. 

Translate  -  analyze  statements  for  interface  consistency 
(proper  decomposition).  Check  to  see  if  problem  is 
defined  as  originally  intended. 

Approve  -  go  through  approval  channel  for  acceptance. 

If  not  finished,  redo  the  process. 


ANSWERS: 


of  Responses 


Response 


55 

Yes 

18 

No 

27 

No  response 

comments:  (Bullets  indicate  individual  responses.) 

•  Yes,  but  Government  lags  in  interviews. 

t  Yes,  the  specs  must  reach  a  complete  and  accurate 
state  and  then  become  baselined  and  controlled 
by  the  customer.  Otherwise  contractor  will  shift 
the  "spec"  to  accomodate  the  "as  built"  system 
which  will  undoubtably  not  satisfy  user  require¬ 
ments,  thereby  entailing  a  redesign  and  costly 
process  to  provide  what  the  customer  really  wants. 
Make  a  commitment  and  then  build  it. 
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SYSTEM  DEVELOPMENT  PROCESS:  Question  6  comments  continued , 


QUESTION  7; 


ANSWERS: 


•  No,  I  use  simulation  at  a  much  earlier  level.  I 
try  to  identify  the  hard  parts  technically  difr 
ficult  problems  and  solve  them  easily  even  though 
they  may  be  at  a  lower  level. 

•  This  appears  to  be  a  top-down,  cooperative  user/ 
developer  scheme  to  arrive  at  a  mutually  accept¬ 
able  design  description.  Hopefully,  there  is 
more  interchange  at  the  "jot"  stage  than  appears 
here.  It  must  not  become  an  independent  effort 
on  the  part  of  the  designer  and  he  must  not  pro¬ 
ceed  so  far  along  as  to  confound  the  user, 

•  Not  really.  May  apply  to  micro  level  where  a  great 
deal  of  freedom  exists  and  task  definition  leaves 
room  for  modification  of  task  (and  commensurate 
modification  of  related  tasks).  At  macro  level, 
this  freedom  does  not  generally  exist,  Customer 
predetermines  much  of  the  information  you  are  look¬ 
ing  for  above. 

•  Yes,  unfortunately! 

•  Yes.  One  item  not  identified  above  I  consider 
extremely  important  prior  to  starting  the  system 
design,  but  included  in  my  concept  of  design  is 
"study  the  big  picture".  Comprehend  where  your 
system  fails  in  the  "total  system"  and  understand 
the  functional  area  of  application.  I  have  found 
this  approach  most  effective  as  it  provides  a 
reference  point  for  the  designer  to  talk  to  the  user 
in  the  users  frame  of  reference  and  not  the  techni¬ 
cal  frame  of  reference  of  the  designer. 


What  techniques  do  you  use  to  make  the  operation  of  your 
facility  more  efficient?  (Respondents  were  asked  to  check 
those  that  appl ied. ) 


%  of  Responses  Techniques 

18  Smart  snapshots  (memory  and  timing) 

45  Detail  of  simulation  modelling 


SYSTEM  DEVELOPMENT  PROCESS:  Question  7  percentages  of  responses  continued , 


%  of  Responses  Techniques 

27  Number  and  detail  of  user  aid  requests 

(e.g,,  trace,  clock  pilot,  print-out 
options) 

18  The  simulation  should  verify  only  the 

module  necessary  to  satisfy  the  objective 

64  Use  real  module  for  performance  testing 

27  Use  "fast"  module  where  logical  verifi¬ 

cation  only  is  required 

18  Order  of  testing  a  module  where  logical 

verification  only  is  required 

45  Use  right  tools  for  the  test  objective 

(e,g,,  analyzer  to  statically  verify 
possible  memory  conflicts,  possible 
priority  conflicts,  timing,  error  re¬ 
covery,  etc.) 

27  When  submitting  new  runs  r  don’t  run 

variants  until  it  is  assured  that  one 
deck  has  run  through  successfully10 

55  Use  support  software  which  monitors 

computer  usage. 


Thirty-six  percent  of  the  participants  did  not  respond. 


question  8.-  Which  tools  and  phases  apply  to  your  target  system  development? 
(Respondents  were  asked  to  check  those  that  applied,) 


answers.-  (The  percentages  are  tabulated  in  the  chart  that  follows,) 

Fifty-five  percent  of  the  participants  did  not  respond 
to  Question  8. 


10  Important 


SYSTEM  DEVELOPMENT  PROCESS:  Question  8  percentages  of  responses 


♦ 


? 


♦ 


COMPONENT  TOOLS 

-  Requirements/Spec¬ 
ification  Language 

-  Analyzer 

-  Static  Resource 
Allocation  Tool 

-  Other 

SUPPORT  TOOLS 

-  Data-base  Structure 

-  Resource  Monitoring 

-  Inter-revision  Updater 

-  Collector 

-  Text  Editor 

-  Text  Formatter 

-  Simulator 

-  Emulator 

-  Performance  Monitor 

-  Other 

INCREMENTAL  TOOLS 

-  Assembly  Language 

-  Macro-processor/ 
Assembler 

-  Higher  Order  Language 

-  Compilers 

-  Structured  Flowcharter 

-  Interactive  Debugger 

-  Interpreter 

-  Other 


» 
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question  9:  Could  some  tools  be  the  same  ones  that  are  not  the  same 
today?11 


%  of  Responses  Response 

9  Yes  • 

91  No  Response 


question  10:  The  following  is  a  list  of  recommendations  we  made  for  develop¬ 
ing  a  system,  (Respondents  were  asked  to  check  those  that  they 
be! ieved  would  be  beneficial.) 


answers :  %  of  Responses  Recommendations 

82  Define  systems  at  front-end  with 

hierarchical  definitions  (integrate 
from  the  beginning) 

82  Perform  design  and  verification  role 

on  all  phases  making  use  of  a  formal 
definition  technique** 

82  Provide  interface  specification  document 

-  common  definition  of  terms 
dictionary 

-  common  data  dictionary 

-  common  structure  dictionary 

-  common  function  dictionary 

64  Provide  user  manual  which  provides  check¬ 

lists  and  explains 

-  how  to  interpret  interface 
specification  document  (for 
users) 

-  how  to  design  modules  for  adding 
to  the  "library"  of  the  inter¬ 
face  specification  document  (for 
designers) 

-  how  to  define  standards  for 
system  development  (for  managers) 


nUnclear  statement:  Do  you  mean  that  an  improved 
tool  replaces  a  less  effective  one?  Or  do  you 
mean  that  a  tool  can  perform  in  multiple  phases, 
that  is,  program  val idation/production? 

[/wr/fOR's  note:  The  latter,] 

12Very  important. 
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SYSTEM  DEVELOPMENT  PROCESS:  Question  10  percentages  of  responses  continued . 


%  of  Responses  Recommendations 

55  Provide  guide  to  implementation 

-  how  to  go  from  specification 
to  computer 

-  how  to  provide  for  reconfigu¬ 
ration  of  functions  in  real  time 

n 

Provide  a  development  model 
Answers:  Others  (specify) 

•  Suggest  alternative  organizational  structures. 
The  process  will  be  no  more  effective  than  the 
array  of  personnel  assigned  and  trained  to 
carry  it  out.  The  finest  development  process 
conceivable  is  worthless  without  competent 
people  to  carry  it  out, 

•  "How  to"  manual  for  the  managers  of  the  require¬ 
ments  determination  organizations, 


question  11:  We  believe  that  to  change  to  techniques  which  will  make  a 

positive  impact  in  system  development,  that  an  initial  invest¬ 
ment  is  necessary  to  define  and  develop  a  general  model  for 
defining  systems.  Would  you  be  interested  in  revamping  your 
own  methods13  u  ? 


answers:  %  of  Responses 

64 

Evolutionary 1 5 

18 

Revolutionary 16 

0 

Not  Certain 

36 

No  Response  1 

13Do  not  have  my  own  methods. 

•••Evolutionary  in  the  course  of  an  on-going  program;  revolutionary 
at  the  outset  of  a  next  project  with  the  qualification  that  the 
methods  have  proved  valid. 

15With  constraints, 

1 6How? 
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question  12:  On  the  next  page  is  a  table  illustrating  the  evolutionary 
nature  of  software,  as  we  see  it,  Where  do  you  think  your 
own  development  techniques  fit  in? 


answers:  (The  bulleted  item  is  an  individual  response.) 


FIRST 

SECOND 

THIRD 

FOURTH 

FIFTH 

OTHER 


•  Third:  there  is  yet  to  be  one  example  of  a 
specification  language  applied  in  the  develop¬ 
ment  of  our  Army  tactical  systems  of  which  I'm 
aware.  We  do  accept  the  fact  that  the  discipline 
afforded,  avoidance  of  semantic  problems,  miscon¬ 
ceptions,  reduction  in  ambiguities,  and  incom¬ 
pleteness,  as  well  as  gains  in  testability  would 
result  from  the  use  of  a  formal  specification 
language;  perhaps  one  that  would  be  a  close  de¬ 
rivative  of  a  requirements  language.  Unfortunately 
the  step  has  yet  to  be  taken  and  we  certainly  won't 
get  to  the  "fifth"  unless  we  move  to  the  "fourth". 


GENERAL  MANAGEMENT  QUESTIONS 

question  i:  Select  a  project  of  reasonable  size  which  you  managed,  Suppose 
you  were  given  a  chance  to  do  it  over  again  any  way  you  wished. 
What  would  you  do  the  same  way?  What  would  you  do  differently? 


answeps:  (The  bulleted  items  on  the  following  page  are  individual 
responses  by  the  participants.) 
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GENERAL  MANAGEMENT  QUESTIONS:  Question  1  responses . 


•  Would  maintain  hard  line  with  requirements  developers, 

Would  use  more  innovative,  state-of-the-art  software 
techniques, 

•  Same;  Modularity,  a  la  Parnes,  of  system  functions  into 

a  set  of  "minimum  essential  functions"  for  phased  develop¬ 
ment. 

Different:  Closer  tie  with  contractor  to  input  "governement 
view"  on  design  options  selected  durinq  develooment. 

•  Haven't  really  managed  one  to  completion  yet. 

•  Keep  competent  people,  Would  want  more  concentration  on 
front-end  planning  and  time  to  test  the  concept  through 
simulation.  More  time  to  organize  the  program  before  it 
becomes  final,  More  and  better  cross  communication  among 
participants.  More  visibility  for  people  to  see  the 
system  take  shape  beyond  their  own  effort, 

•  Have  a  non-computer  specialist  (management,  finance,  etc,) 
act  as  co-project  engineer, 

•  Firmer  requirements/more  simulation  and  closer  contract 
monitoring/testing.  Use  HOL/imorove  development  and  test 
environment. 

•  Coding  would  be  the  absolute  last  thing  to  be  done.  Descrip¬ 
tive  flows  would  be  complete  and  all  "loose"  ends  tied  off 
and  squared  away.  Structured  walk  throughs.  Establish 
reasonable  B-level  descriptions  and  maintain  CM  baseline. 

•  Not  in  the  way  I  managed  it,  but  in  the  way  it  was  managed. 

•  Very  little  in  the  same  way.  Would  start  with  better 
definition  of  requirements.  Started  with  poor  A-level 
specifications,  kept  changing  as  system  grew.  Nature 

of  R  &  D  is  2  steps  up  and  one  back  and  so  in  some  sense  wasteful. 
Would  verify,  change  acquisition  process  to  have  prime 
bidder  demonstration  and  auditor  for  software.  Biggest 
thing,  don't  try  to  buy  it  cheap.  In  both,  cases  the 
Government  allows  contractors  to  "buy  in"  (i.e., 
a  contractor  estimate  less  than  government  estimate  should 
be  suspected). 

question  2:  What  criteria  do  you  use  to  know  what  constitutes  a  phase  of 
development?  (For  example,  how  do  you  know  if  a  phase  is 
complete?) 


answers:  (The  bulleted  items  on  the  following  page  are  individual 
responses  by  the  participants.) 


GENERAL  MANAGEMENT  QUESTIONS:  Question  2  responses. 


•  Primarily  use  testinq  against  specific  criteria  on 
definable  products  (i,e,,  modules,  programs,  subsets, 
system), 

$  Minimum  essential  set  of  functions,  Time/Cost, 
Demonstra table  performance, 

•  Milestone  accomplishment.  Predefine  what  tasks  ape 
to  be  completed  during  a  phase  along  with  criteria 
to  determine  "completion"  (satisfactorily), 

•  We  are  prone  to  carefully  set  key  events  or  milestones 
signifying  an  interim  completion  point  (i,e,,  initial 
test  of  edit  program).  Where  we  stumble  is  how  do  we 
validate  the  test,  how  do  we  know  it  was  successful 
enough.  Conclusion,  we  are  sloppy  on  our  metrics,  I 
try  to  use  some  objective  measure. 

•  We  define  deliverables  and  review  them  when  complete, 

•  If  the  performances  and/or  functional  requirements 
have  been  satisfied,  then  the  activity  is  complete  for 
the  overall  system, 


question  3:  What  are  the  most  important  characteristics  you  look  for  in 
the  people  you  hire?  (Some  respondents  identified 
characteristics  in  order  of  importance.) 


ANSWERS: 


Responses  by 


%  of  Responses 

Characteristics 

mportance 

64 

Intelligent 

2 

_4_ 

_2_ 

64 

Motivated 

1 

JL 

18 

Educational  background1  2 

5 

_5_ 

0 

Attractive  personality 

3 

64 

Experience 

3 

Answers : 

Other  (explain) 

•  Ability  to  work  in  tribe  environment, 

•  Character  (Number  1),  i.e.  reliability,  honesty, 
dedication,  etc.  An  intelligent,  motivated 
individual  gains  experience  and  learns  quickly. 
I'm  becoming  less  impressed  with  an  individuals 
experience, 

•  Doer,  gets  the  job  done  with  independence, 


depending  on  position  -  intelligent  and  motivated  may  be  more  trainable 
and  productive  than  an  individual  with  educational  background. 

2Most  people  in  this  business  have  good  credentials.  More  important  at  the 
beginning  of  a  career,  then  other  factors  begin  to  take  on  more  signif¬ 
icance. 
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QUESTION  4: 


ANSWERS : 


QUESTION  5: 


ANSWERS: 

•  None 

9  Standardized  documentation;  systems  analysis  documentation, 
test  packages.  None  automated, 

•  Modularity  of  functions,  standardized  software  documentation, 
standardized  languages/support  software, 

•  Limited  standardization  -  specification  testing, 

•  Limited  automated-testing 

•  Improvement  in  use  of  higher  order  languages,  test  techniques. 

We  are  better  at  handling  the  period  from  the  midpoint  of  the 
project  to  the  end  than  from  the  beginning  to  the  midpoint. 

•  Requirements/Design  -  formal.  Configuration  control  -  automatic. 

•  A,B,C  levels  and  testing.  Automatic  verification  system  tool 
DoD  5000.29,  5000. 31s. xx,  DoDI. 


question  6:  In  what  ways  have  you  and  your  people  advanced  the  state- 
of-the-art  in  various  technology  areas? 


answers.-  (Bulleted  items  are  individual  responses  by  participants.) 

•  Parnas  technology,  HDM,  PSL/PSA,  H0S  (a  little),  SDL, 
Software  Engineering,  standard  hardware/ interfaces, 
protocols, 

9  Hardly  any. 


Are  you  able  to  envision  your  own  system  development  process 
as  being  systematic? 

%  of  Responses  Response 

55  Yes34 

9  No 

36  No  Response 

What  tasks  in  your  project  have  been  converted  from  ad  hoc 
methods  to  standardized  methods?  Which  have  been  automated? 


(Bulleted  items  are  individual  responses  by  the  participants.) 


3If  left  alone! 

“Hopefully.  Do  you  mean  my  contractor's  or  what  we  specified? 
[Author's  note:  the  latter.] 
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GENERAL  MANAGEMENT  QUESTIONS :  Question  6  responses  continued . 


•  Good  at  determining  a  better  way  to  carry  out  development 
programs  but  poor  at  putting  it  into  practice. 

•  Code  generation,  software  verification  and  test  tpol 
development,  software  management, 

•  Awareness,  HOS  contract,  CENTACS  program, 


question  7:  What  would  you  like  to  do  next  with  respect  to  technology 
advancement? ' 


answers:  (Bulleted  items  are  individual  responses  by  the  participants.) 

•  Develop  a  means  for  assessing  software  reliability, 

•  Standardized  architecture  with  certification  techniques 
to  allow  the  purchase  of  "systems"  vs.  computers. 

•  The  variety  of  software  tools  available  boggle  the  mind. 

Would  like  to  be  involved  in  an  effort  to  integrate  a 
few  into  an  orderly,  logical  process  applicable  to  a 
real-world  system  development, 

•  Improve  integration  of  requirements  and  design. 

•  Technology  insertion  via  MCF,  DoD-1  convergence,  Production 
of  OS,  support  software  tools, 


question  8:  What  properties  do  you  want  your  system  to  have?  (Respondents 
were  asked  to  number  in  order  of  importance.) 


ANSWERS: 


Responses  by 

Properties  Importance 


Modular5 6 

T 

T 

1 

T 

5 

2 

Reliable 

2 

1 

1 

2 

1 

1 

1 

Efficient 

1 

4 

5 

6 

1 

7 

Easy  to  understand6  7 

6 

5 

1 

3 

1 

5 

Traceable7 

8 

3 

2 

1 

6 

5Under  broader  category  of  extendable. 

6Use  ninth-grade  maintenance  and  user. 

7These  should  be  inherent,  cannot  rate  with  others. 


GENERAL  MANAGEMENT  QUESTIONS:  Question  8  responses  continued , 


Properties 

Transportabl e7 
Interoperabil i ty 8  9 
Flexible5 
Other  (explain) 

•  Correct10 

•  Maintainable 


Responses  by 
Importance 


7 

2 

1 

8 

1 

8 

4 

8 

1 

7 

4 

5 

7 

1 

4 

4 

1 

3 

1 

3 

question  9:  a)  What  are  your  greatest  costs?  b)  Do  you  know  how  to 
reduce  them?  c)  Do  you  want  to  reduce  them? 

answers:  (Bullets  indicate  individual  responses  by  the  participants.) 


A 

B 

c 

Correct  implementa¬ 
tion,  T&E,  verifica¬ 

•  Support  enhancement. 

•  Of  course. 

tion. 

Peopl e 

•  No,  other  than  improv¬ 
ing  their  productivity. 

t  Need  to  reduce  time 

•  Yes 

People  over  long 

factor.  Need  shorter 

periods  of  time. 

development  process. 

Can  do  this  by  increasing 

Personnel 

productivity  of  people. 

We  must  get  a  better 

Software  people; 
labor  intensive  but 

product  earlier. 

lousy  productivity. 

•  More  automation. 

9  Bring  about  discipline 
as  per  hardware  process. 

®Difficu1t  question:  specific  system  considered  does  not  have 
interoperability  reg.:  low  priority  on  that  item. 

9  If  required,  then  part  of  correct. 

“That  is,  it  does  what  it  is  supposed  to  do. 


QUESTION  10 

ANSWERS 

§ 

QUESTION  11 

ANSWERS 


What  are  the  relative  costs  involved  in  your  project 
development?  For  example,  what  percentage  of  the  costs 
are  spent  on  verification?  On  design? 


:  (Bulleted  items  are  individual  responses  by  the  participants,) 


Design  and  coding  highest;  verification  least; 
integration  and  testing  -  median. 

Definition,  15%;  Design,  15%;  Implementation,  15t20%; 

T&E,  15%;  Support  40%.  (Including  upgrade,  etc. 
correcting  design  faults.) 

Cannot  break  out  development;  verification  =  ,v30%  of 
design  development. 

Little  is  spent  on  design  or  concept  validation  -  unless 
it  is  a  software  driven  test  bed,  Most  of  the  invest¬ 
ment  is  in  system  development,  hardware  build,  and  test¬ 
ing. 

Requirements,  10%;  Design,  20%;  Build,  10%;  Verification/ 
Documentation,  60%, 

<50%  Verification,  Integration,  Testing;  >10%  Requirements; 
=30-40%  Design/Cost/Debug, 


How  much  effort  is  involved  in  training  personnel11? 
(Respondents  were  asked  to  check  those  that  applied,) 


%  of  Responses  Training  Methods 


73 

Special  courses 

36 

In-house  seminars 

9 

Invited  speaker  seminars 

45 

Technical  exchanges 

9 

Other  (explain) 

•  On  the  job  training 

u 


Not  enough, 


question  12:  What  are  the  functions  you  perform  (and  in  what  order) 

to  continuously  improve  your  project  development  process12? 
(Respondents  were  asked  to  check  those  that  applied.) 


ANSWERS: 


%  Of 

Responses 


45 

Periodic  reviews 

45 

Meetings 

36 

Memo  series14 

27 

Systematic  approval 

mechanisms 

Answers : 

Other  (explain) 

Responses  by 
Importance13 


3 

2 

1 

1 

2 

1 

1 

2 

3 

1 

4 

4 

4 

4 

2 

3 

3 

2 

3 

•  Show  personal  interest  in  personnel  by  being 
visible  in  their  area,  asking  pertinent  but 
not  antagonistic  quesitons,  showing  interest. 


question  13:  If  you  had  to  choose  an  absolute  goal  as  a  manager,  what 
would  it  be? 


answers:  (Bulleted  items  are  individual  responses  by  participants.) 

•  Satisfy  the  user! 

•  Abolish  questionnaires  that  use  essay  questions. 

a  On  time,  under  budget,  totally  acceptable  product 
to  the  user. 

a  To  create  an  environment  in  which  personnel  can 
work  productively. 

a  Delivery  on  time,  within  cost,  of  a  workable  and 
supportable  system. 

a  Be  rich! 

a  Have  reasonable  latitude  in  design  decisions. 

a  To  win  customer  approval,  corporate  favor,  an 
expanding  business  base. 

a  Make  sure  everyone  is  productive  all  the  time, 
using  available  tools  and  techniques  through 
training  and  on-the-job  execution. 


^Should  change  "development"  to  "management". 

uSome  respondents  prioritized  their  aswers  as  shown  here. 

‘‘‘Decision  logs;  not  automated. 
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CHARACTERISTICS  OF  A  MANAGER 


question  l:  Rate  the  following  on  a  scale  of  1  to  18  (where  181  is 

the  highest  priorty)  the  importance  of  personality  traits 
for  a  manager. 


answers :  Personality  Traits 


Responses  b.y  Importance 


Sense  of  humor 

0 

4 

16 

9 

12 

13 

4 

A 

5 

Humble 

3 

5 

17 

6 

14 

17 

3 

B 

3 

Confident2 

17 

10 

14 

7 

18 

/ 

10 

13 

B 

10 

Flexible  to  change 

15 

9 

6 

13 

18 

/ 

3 

16 

15 

Fair3 

8 

14 

3 

5 

18 

/ 

7 

11 

C 

7 

Compassionate4 

4 

6 

12 

3 

18 

19 

6 

B 

12 

Understands  people 

9 

1 3 

2 

10 

18 

/ 

5 

18 

11 

Technically  competent 

18 

12 

9 

14 

12 

✓ 

6 

12 

D 

4 

Awareness5 

10 

8 

8 

18 

14 

4 

8 

6 

Possitive  attitude 

14 

16 

5 

8 

14 

/ 

1 

17 

14 

Decisive6 

11 

17 

7 

17 

16 

/ 

12 

10 

B  1 

17 

Delegation  of  responsibility 

13 

11 

4 

16 

18  ! 

/ 

11 

15 

/ 

16 

Aloof 

5 

2 

18 

1 

1 

16 

2 

E 

Integrity 

16 

18 

1 

15 

18 

✓ 

2 

14 

18 

Discrete 

6 

7 

13 

2 

17 

19 

5 

B 

9 

Loyalty7  8 

7 

15 

11 

11 

18 

j 

14 

7 

A 

13 

Pragmatic 

12 

3 

10 

12 

18 

/ 

20 

9 

F 

8 

Other 

1 

j 

•  Systematic 

4 

1 

•  Managerial ly  competent 

14 

•  Focus9 

17 

•  Organizational  ability 

19 

•  Written  communication  skills 

9 

•  Oral  communication  skills 

12 

backwards , 

2Too  much  can  hurt. 

Essential . 

40esirable,  not  essential, 

5(or  lack  thereof)  of  your  own  capabilities, 
6Nice,  if  got  others. 

7Up  and  down  both. 

8With  objectivity. 

9At  high  level  problems  -  not  details. 


KEY 


a  -  Top  Middle 
B  -  Low 
c  -  High 
d  -  Middle 
e  -  Last 
f  -  Very  High 
/  -  Neccessary. 
but  not 
prioritized 


QUESTION  2: 

What  incentives  do  you  provide  for  people?  (Respondents 
were  asked  to  check  those  that  applied,) 

ANSWERS : 

%  of  Responses 

Incentives 

55 

Emphasis  on  merit  increases 

73 

Compliments  for  good  work 

9 

Optional  tools  to  avoid  drudgery 

82 

Encouragement  to  be  creative 

55 

Promotions 

36 

Interpretation  of  organizational 
position 

Aswers : 

Other  (explain) 

•  New 

titles?! 

QUESTION  3: 

What  enforcements 
(Respondents  were 

do  you  execute  with  respect  to  people? 
asked  to  check  those  that  applied.) 

ANSWERS: 

%  of  Responses 

Enforcements 

18 

Set  working  hours 

36 

Set  techniques  or  topis 

36 

Position  in  personnel  hierarchy  is  maintained 

55 

Working  relationships  are  maintained19 

Answers : 

Other  (explain) 

•  Flexible  time 

•  Deliverable  goals 

•  Mission  accomplishment,  responsible  for 
actions,  resource  accountability 

•  No  set  working  hours 

•  Negotiate  work,  objectives  and  review 

question  4:  What  tradeoffs  do  you  make  with  respect  to  reliability  and 
cost  effectiveness? 


answers:  (The  bulleted  items  on  the  following  page  are  individual 
responses  by  the  participants,) 


10But  flexible. 
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CHARACTERISTICS  OF  A  MANAGER:  Question  4  responses. 


•  In  many  military  (tactical)  systems,  reliability 
is  dictated  and  the  tradeoff  is  not  an  option, 

•  Reliability  can  be  traded  off  for  cost  only  after 
availabil ity  to  accomplish  the  military  mission 
is  satisfactory. 

•  Under  design  to  cost,  a  certain  threshold  of  per¬ 
formance  and  reliability,  beyond  that  is  frosting 
subject  to  priority  deletion  by  cost  factors. 

•  Systems  analysis  at  start  of  project. 

9  Reliability  70%. 


question  5:  Select  a  system. 

(Ninety-one  percent  of  respondents  did  not  respond  to 
this  question.) 

a)  List  all  of  the  systems  that  exist  within  the  environment 
of  that  target  system  and  rate  them. 


SYSTEM 

HELPFUL 

HINDRENCE 

OBSOLETE 

NECESSITY 

HOL 

/ 

/ 

Simulator 

/ 

Assembly 

/ 

/ 

b)  What  new  systems  would  be  desirable  to  have  exist  within 
the  environment  of  that  target  system? 


Answer:  Off-the-shelf  "standardization"  products, 
(i.e.,  computers,  peripherals,  support 
software,  GFE  products). 


QUESTION  6  a 
b 
c 
d 
e 
f 


How  many  people  interface  with  you  above  your  level? 
On  the  same  level? 

Working  for  you? 

How  many  people  work  for  you  in  total. 

Do  some  of  these  interfaces  conflict  with  each  other. 
Are  some  non-existent? 


[Author's  note:  Responses  to  this  question  kept  proprietary,] 
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question  7a:  How  closely  does  your  personnel  structure  correspond  to 

your  project  and  project  development  structures?  (Respondents 
were  asked  to  check  one  that  most  applied.) 


Not  related 
Related 
Hard  to  tell 

b:  Explain. 


answers.-  (Bulleted  items  are  the  individual  responses  of  participants.) 

•  Separation  of  analysts,  programmers,  and  tied 
together  by  technical  support  as  independent 
evaluator. 

•  Usually  a  project  will  be  staffed  by  drawing 
resources  from  various  elements  within  the 
center.  Matrix  management. 

•  Contractors  segmented  into  phases  and  specific 
capabilities  for  system  development  in  the  whole. 

•  We  are  a  matrix  organization  so  my  organization 
has  several  people  working  on  different  projects. 

•  Separately  into  functional  groupings. 


question  8:  Do  you  have  a  means  of  integrating  the  various  people 
functions  as  well  as  reviewing  the  performance  of  your 
people  that  you  consider  to  be  successful? 


answers:  %  of  Responses 


Responses 


9  Reasonably  so 

9  Not  effectively 

9  No 

Explanations:  (Bulleted  items  are  individual  responses  by  participants.) 

•  Yes,  establish  teams  of  various  resources  to 
accomplish  given  task  based  on  areas  of  expertise. 

$  Yes,  we  can  shift  people  into  various  roles  for 
the  benefit  of  the  individual  and  the  job  to  be 
done. 

•  Cross-fertilize  and  cross  track.  Back  up  capability. 

question  9:  Do  you  have  checklists  for  yourself  and  your  people.  If  so, 
what  are  they? 


answers:  %  of  Responses 


Response 


No  response 


•  A  very  formal  process  of  documenting  performance 
on  an  appraisal  form.  Must  describe  technical 
competence,  verba  1 /wri tten  communications  ability, 
cooperation,  etc.  Must  recommend  further  training, 
ability  to  perform  and  level  of  performance  on 
assigned  job,  etc. 

•  Standard  Government  issue,  performance  appraised 
and  job  description  tools. 

•  Acquisition  procedures. 
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APPENDIX  I:  ACQUISITION  QUESTIONNAIRE 


PART  2* 


*Part  2  responses  have  not  been  summarized  here.  A  follow-on 
to  this  effort  would  make  use  of  the  responses  to  Part  2. 
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FOUNDATION  FOR  FORMAL  METHODS 


T.  Suppose  you  were  asked  to  list  a  set  of  general  principles  to  adhere  to 
in  defining  a  system.  What  would  they  be?  _ _ 


2.  Could  rules  be  derived  from  these  principles? 


3-i  How  would  you  train  others  with  respect  to  your  principles  and  rules? 


4.  Can  you  think  of  a  communication  problem  where  an  object  is  confused  with 
its  name?  What  is  it?  _ 


5 


♦FOUNDATION  FOR  FORMAL  METHODS  (continued) 


5.  Can  you  think  of  a  communication  problem  where  an  object  was  confused 
with  the  definition,  implementation,  description  or  execution  of  that 
object?  What  was  it?  _ 


6.  What  ways  do  you  abstract  with  respect  to  system  definitions  for  people, 
hardware,  etc?  Are  they  standard?  _ 


7.  How  do  you  share  common  tools,  common  modules,  comnon  expressions,  etc.? 
Check  those  that  apply. 

_  Library 

j  _  Word  of  mouth 

1  _  Common  manager 

'  _  Other  (explain) _ 


8.  What  objects  do  you  describe  when  defining  a  system?  What  characteristics 
about  these  objects  are  important  to  describe?  _ 

i  _ _ _ 


9.  How  much  emphasis  is  placed  on  powerful  but  simple  notation  techniques? 


*FOUNDATION  FOR  FORMAL  METHODS  (continued) 


10.  Do  definitions  include  control  considerations?  Check  those  that  apply. 

_  Access  rights 

Functions  to  be  invoked 

_  Data  flow 

Orderi ng 

_  Error  detection  and  recovery 

__  Other  (explain) _ 


11.  Specifically,  do  definitions  address  real-time  considerations  (e.g., 
timing,  priorities)?  _ 


12.  Are  you  personally  involved  in  defining  requirements?  What  type  of  logic 
patterns  do  you  and  those  you  communicate  with  use  (e.g.,  a  decision)? 


13.  What  operations  do  you  use  over  and  over  again  on  a  current  project 
(e.g.,  a  particular  form  of  message  processing)?  _ 


14.  What  functional  processes  would  be  beneficial  by  freezing  them  for  later 
use  or  reconfiguration?  _ 


♦FOUNDATION  FOR  FORMAL  METHODS  (continued) 

15.  What  data  types  do  you  commonly  refer  to  (e.g.,  a  particular  message 
format)?  _ 


16.  What  systems  might  be  used  for  more  than  one  application,  computer,  or 
reconfiguration? _ 


17.  What  types  of  information  should  be  recorded  for  later  use?  Check 
those  that  apply. 

_  Errors 

_  Complaints 

_  Wish  lists 

_  Successes 

_  Result  of  tests 

_  Manual  procedures 

_  Other  (explain) _ 


♦REQUIREMENTS  CHARACTERISTICS 


w 


*REQUIREMENTS  CHARACTERISTICS  (continued) 

2.  When  does  a  conceptual  idea  (or  cloud)  become  a  requirement? 


3.  How  do  you  distinguish  requirements  from  specifications? 


4.  What  categories  of  requirements  do  you  have  in  your  system?  How  are 
they  partitioned  with  respect  to  resources?  _ 


5.  When  do  you  distinguish  hardware  from  software  from  users  functions? 
How  do  you  distinguish  these  functions? _ 


6.  How  dependent  are  applications  requirements  from  target  machines  and 
target  systems?  _ 


7.  Do  requirements  address  memory  and  timing  applications?  How  are  these 
estimates  made? _  _  _  _  _ 
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REQUIREMENTS  CHARACTERISTICS  (continued) 


8.  For  software  requirements,  how  are  applications  modules,  systems  soft¬ 
ware  modules,  and  system  support  tools  differentiated?  _ 


9.  Do  requirements  address  error  detection  and  recovery?  Do  they  consider 
redundancy  back-up  systems?  How?  _ 


10.  Is  there  an  attempt  to  save  memory  and  time  resource  during  the  require 
ments  phase? _ 


11.  What  type  of  information  is  included  in  a  requirement?  Check  those  that 
apply. 

.  _  Functions 

_  Structures 

_  Data  types 

Other  _ 


12.  What  level  of  detail  do  you  use  to  describe  requirements? 

_  English 

_  Flowcharts 

_  Requirements  language 

_  Block  diagrams 

Other  (specify) _ ] _ ■ 


* REQUIREMENTS  CHARACTERISTICS  (continued) 

13.  How  are  interfaces  to  other  requirements  defined? 


14.  How  are  inputs  and  outputs  characterized? 


15.  How  are  constraints  characterized? 


16.  Are  both  nominal  and  off-nominal  conditions  for  the  operation  of  a 
system  defined?  _ 


17.  Are  restrictions  imposed  on  and  imposed  by  operational  techniques, 
hardware  systems,  software  systems,  checklists,  users,  etc? _ 


18.  What  type  of  protection  is  incorporated  in  the  system  from  human  errors? 


19.  Are  effects  of  each  module  with  respect  to  simulators,  users,  etc 
recorded?  _  _  _ 


‘REQUIREMENTS  CHARACTERISTICS  (continued) 

20.  Are  hardware  features  or  system  software  features  available  which  aid 
in  the  definition  of  requirements?  _ 


21.  What  differences  exist  between  the  host  environment  and  target  environ¬ 
ment  that  affect  the  requirements?  _ 


22.  How  are  your  requirements  originally  created  (e.g.,  the  existence  of 
a  threat)?  How  many  groups  of  people  and/or  organizations  are  in¬ 
volved  in  defining  the  requirements?  _ 


LIFE-CYCLE  ASPECTS  OF  SYSTEM  DEVELOPMENT 


1  .  Do  the  same  people  implement  the  requirements  th3t  design  the  require¬ 
ments?  _ _ 


2.  Is  there  an  official  review  process  for  the  integration  of  requirements? 


Can  you  track  the  history  of  a  requirements  change? 


LIFE-CYCLE  ASPECTS  OF  SYSTEM  DEVELOPMENT  (continued) 


4.  Can  you  track  the  history  of  an  anomaly? 


5.  What  happens  to  an  error  when  it  is  too  late  to  fix  the  requirements? 


6.  What  impacts  are  considered  for  each  requirement  or  requirement  change? 

Check  those  that  apply. 

_  Support  tool  change  (e.g.,  simulator) 

_  Personnel  training  change 

_  Mission  change 

_  Job  security 

_  Schedules 

_  Other  (specify) _ _ 

_  I 

7.  Are  support  systems  changed  to  correspond  with  relevant  system  changes? 


8.  Are  requirements  changes  or  errors  traced  for  second  and  third  order 
effects  in  a  system?  _ 


9.  What  types  of  development  plans  and  milestones  are  made  for  requirements 
definitions? 

_  Customer  review 

_  In-house  reviews 

_  Acquisition  checkpoints 

_  Other  (specify) _ 


LIFE-CYCLE  ASPECTS  OF  SYSTEM  DEVELOPMENT  (continued) 


TO.  Are  standard  forms  used?  Check  those  that  apply. 

New  requirements 
Changes  to  requirements 
Errors  reported 
Others 


*11.  What  is  the  approval  hierarchy  and  approval  focal  point  for  new 

requirements,  changes  to  requirements,  and  anomaly  fixes?  Is  there 
a  central  clearing  house?  Are  there  official  sign-offs?  Is  there 
a  numbering  system?  When  does  it  occur  in  the  life  cycle.  _ 


4 


m 


*12.  What  are  the  various  technical  responsibilities  and  technical  disciplines 
involved  in  the  requirements  phase?  _ 


13.  Is  there  a  method  for  not  letting  requirements  "slip  through  the  cracks?" 


14.  Are  there  methods  for  tracking  changes,  anomalies,  fixes  throughout  the 
system,  support  systems,  and  users  of  the  system?  _ 


LIFE-CYCLE  ASPECTS  OF  SYSTEM  DEVELOPMENT  (continued) 


15.  Are  there  methods  for  initiating  action  items  and  the  closing  of  these 
action  items? 


16.  Are  there  methods  of  introducing  improvements  into  the  requirements  phase 
as  a  result  of  previous  problems  found?  _ 


17.  Is  there  any  attempt  to  make  common  use  of  modules  in  various  parts  of  a 
system?  Are  common  modules  used  in  systems  and  their  respective  support 
systems,  or  is  this  considered  a  problem  of  generic  errors?  _ 


18.  Are  specifications  checked  to  see  if  they  meet  the  requirements? 


19.  How  far  through  the  development  process  is  a  requirement  monitored  for 
consistency?  _ 


20.  Is  there  independent  verification  of  requirements? 
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LIFE  CYCLE  ASPECTS  OF  SYSTEM  DEVELOPMENT  (continued) 

21.  Are  there  methods  for  publishing  vital  and  fast  turn-around  information 
to  users  of  a  system?  _ 


22.  What  is  the  education  and  the  background  of  the  engineers  and  management 
involved  in  the  design  of  the  requirements?  Give  percentages. 

_ %  of  Phds 

_ %  of  Masters 

_ %  of  Bachelors 

_ _ %  of  15  -  20  years  experience 

_ %  of  10  -  15  years  experience 

_ %  of  5  -  10  years  experience 


23.  What  background  mix  is  most  advantageous? 
Explain  _ ‘ 


AI-68 


TOOLS  AND  TECHNIQUES1" 


1.  Software  tools  available  in/on  the  data  processing  system(s)  used  by 
your  installation:  Check  those  that  apply. 


Automated  documentation 

Source  text  manipulation 

Program  optimization 

Aids  built  into  compilers 

Special  programming  lan¬ 
guages/compilers 

Preprocessors 

Program  performance 
evaluation 

Requirements/specification 

languages 

Others  (specify) 


"''Excerpts  taken  from  Computer  Software  Review:  The  Use  of  Tools  and 
Techniques,  United  States  General  Accounting  Office. 
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TOOLS  AND  TECHNIQUES  (continued) 


2.  Software  techniques  inyour  installation:  Check  all  items  that  are 
true  in  the  matrix  below. 


Code  arrangement 
Descriptive  documentation 
Performance  documentation 

Embedded  documentation 

Programming  practices/ 
standards 

Re-use  of  already  written 
code 

Quality  assurance  organiza¬ 
tion/management 

Requirements/specifi cation 
standards 

Programming  organization/ 
management 

Others(s)  (specify) 


TOOLS  AND  TECHNIQUES  (continued) 


3.  Does  your  installation  have  formal  written  rules  or  standards  for 
Check  those  that  apply. 

Acquisition  of  software  tools? 

___  Development  of  software  tools? 

Use  of  software  tools? 

_  Acquisition  of  software  techniques? 

_  Development  of  software  techniques? 

_  Use  of  software  techniques? 

_  Evaluation  of  effectiveness  of  tools  after  use? 

_  Evaluation  of  effectiveness  of  techniques  after  use? 


Please  complete  the  sentences  below  by  checking  one  of  the  listed 
options.  Check  only  one  for  each  statement. 


/  *  / 


iff 


Cost/benefit  studies  are  required 
for  the  acquisition/development  of 


Before  general  adoption,  pilot 
projects  are  required  to  evaluate 
the  benefits  of 


TOOLS  AND  TECHNIQUES  (continued) 


5.  Please  indicate  the  benefits,  if  any,  to  your  installation  of  each  of 
the  listed  software  tools.  Check  those  that  apply. 


TOOLS  AND  TECHNIQUES  (continued) 

6.  Please  indicate  the  benefits,  if  any,  to  your  installation  of  each  of 
the  listed  software  techniques.  Check  those  that  apply. 
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TOOLS  AND  TECHNIQUES  (continued) 


7.  Please  estimate  the  benefits  for  the  new  software  tools  and  techniques 
your  organization  has  adopted  i_n  the  last  four  (4)  years.  Check  one 
column  for  each  item. 


laproveaaat  la 
progranaar  pro¬ 
ductivity  in 
aoftvara  deve¬ 
lopment  (Reduced 
davalopmant  coat) 


Improvement  la 
projraaoer  pro¬ 
ductivity  in 


£7  CJ  £J  CJ  C  £7 


aoftvara  maia- 
tananca  (raducad 
maintenance  coat) 


Raducad  overall 
development  coat 


C  C  C  CJ  lJ  CJ 


Raducad  ovarall  £J  £~J  j~J  /  J  ,~J  j—j 
aiaintanaoca  coat  —  1 —  1 —  L —  - 


Othar  (Spacify)  CJ  C  C  CJ  C  C 


8.  If  benefits  have  been  obtained  from  the  use  of  either  tools  or 
techniques,  have  they  been  documented  in  a  formal  summary  report? 

_  1-  Yes 
2.  No 


9.  If  a  formal  report  exists  (yes  to  #8),  will  your  installation  share  it 
with  us? 


TOOLS  AND  TECHNIQUES  (continued) 


10.  Tools  portability:  To  the  best  of  your  knowledge,  which  of  the  items  in 
the  matrix  below  apply  to  the  tools  now  present  in  your  installation? 
Check  those  that  apply. 


Automated  documentation 

Source  text  manipulation 

Program  optimization 

Aids  built  into  compilers 

Special  programming 
languages/compilers 

Preprocessors 

Program  performance 
evaluation 

Req'ts/Spec.  language 
Other(s)  (specify) 


TOOLS  AND  TECHNIQUES  (continued) 


11.  Techniques  portability:  Please  check  the  items  in  the  matrix  below 
that  apply  to  the  techniques  now  used  at  your  installation. 


Code 

arrangement 

Descriptive 

documentation 

Performing 

documentation 

Embedded 

documentation 

Programming 

practices 

standards 

Re-use  of  already 
written  code 

Quality  assurance 

organization 

management 

Requirements/ 

specification 

standards 

Programming 

organization/ 

management 

Other(s) 

(specify) 


i 


TOOLS  AND  TECHNIQUES  (continued) 


12.  What  is  your  installation's  preferred  source  of  software  tools? 
Check  one. 

__  We  use  hardware-vendor-supplied  tools  only. 

We  prefer  to  buy  tools  (over  and  above  hardward- 
vendor-suppl ied)  from  external  sources  off  the  self. 

_  We  prefer  to  have  software  tools  (over  and  above 

hardware-vendor-supplied)  custom-built  by  external 
sources. 

We  use  or  would  use  any  of  the  above-named  sources 
depending  on  the  situation. 

_  Other  (specify) _ 


♦ANALYSIS 


1.  What  are  the  major  categories  of  anomalies  discovered  in  requirements 
definition?  Estimate  relative  importance  of  each  of  the  following 
categories  (in  addition,  add  categories  missing  here): 

_  Incorrect  requirements 

__  Incorrect  change  made  to  requirement 

_  Deficient 

_  Inconsistent 

_  Not  interpreted  correctly 

_  Clerical  translation 

_  Unclear 

_  Did  not  plan  for  system  constraints 

_  Omission 

_  Poor  philosophy 

_  Not  flexible 

__  Other  (specify) _ 


ANALYSIS  (continued) 


2.  How  is  an  error  defined? 


a)  What  is  a  software  error? 


b)  When  is  an  error  an  error  from  a  practical  point  of  view? 


c)  If  a  computer  program  "doesn't  work"  because  of  a  wrong  specification, 
is  the  computer  program  in  error?  _ 


d)  If  an  error  is  officially  known  before  flight,  but  an  official 

decision  is  made  not  to  fix  it,  is  it  an  error  if  it  occurs  during 
flight?  _ 


e)  If  the  specification  is  incorrect,  but  the  computer  program  is  not, 
who  is  in  error?  _ _ 


f)  If  there  is  an  error  in  the  input  to  the  computer  program,  is  the 
computer  program  in  error  if  problems  result  from  that  input?  _ 


g)  Who  is  responsible  for  error  detection  and  recovery,  if  there  is 
such  a  thing?  _ 
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ANALYSIS  (continued) 


h)  If  an  error  is  detected  and  recovered  from,  is  there  an  error? 


i)  If  two  errors  cancel  each  other,  is  there  an  error? 


j)  Is  "better  the  enemy  of  good"  in  providing  for  protection  against 
errors? 


k)  When  more  than  one  specification  exists,  and  they  differ,  which  one 
is  in  error? 


1)  If  there  are  many  errors  and  they  have  one  root  source,  how  many 
errors  are  recorded; 

3.  How  are  most  anomalies  found?  Check  those  that  apply.  List  percentages 

__  Simulation  % 

_  Eyeball  % 

_  Independent  verification  and  validation  _ % 

_ Dynamic  execution  %  ■ 

Other  (specify)  _ 


ANALYSIS  (continuation) 


4.  How  are  problems  categorized  with  respect  to  the  user?  Check  those  that 
apply. 

_  Catastrophic 

__  Worrisome 
_  Annoying 

_  "Funny  little  things" 

_  Borderline 

_  Nonexistent 

_  Known  before  occurrence,  but  forgotten 

_  Known  before  occurrence,  but  not  all  ramifications 

are  known 

___  Other  (specify) _ _ 


5.  Which  types  of  problems  are  not  fixed? 


6.  Who  finds  the  errors?  Check  those  that  apply. 

.  _  Individual  module  engineers 

_  Special  debugging  engineers 

•  Management 

_  Systems  engineers 

_  Other  (specify)  _ 


7.  How  many  errors  are  interface  errors? 


ANALYSIS  (continued) 


9.  What  happens  if  requirements  are  known  to  use  up  too  many  system 
resources?  Check  those  that  apply. 

___  Software  compromised 

_  New  processing  capability  added  to  system 

_  Requirements  deleted 

_  Performance  demands  minimized 

_  Systems  software  changed 

_  Software  tools  (e.g.,  compiler) 

_  Different  tools  used  (e.g.,  language) 

_  Software  converted  to  microcode 

_  Recode  implementation  of  previous  requirements 

Restrictive  requirements 

Other  (explain) _ 


10.  In  what  way  have  software  shortcomings  (e.g.,  an  obsolete  software 
tool)  influenced  requirements  and  changes  to  requirements?  Include 
resources  (time  and  memory)  and  hardware  architecture  considerations. 


♦ 


11.  What  back-up  capabilities  exist  in  case  of  generic  system  errors? 


t 


12.  What  is  done  when  a  system  and  its  support  system  disagree? 
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ANALYSIS  (continued) 


13.  How  often  do  requirements  change?  Check  the  one  which  most  applies. 

_  Very  often 

_  Often 

_  Not  often 

_  Never 

14.  Have  requirements  been  known  to  be  wrong  due  to  incorrect  rumors  or 

assumptions?  _ 


15.  How  is  proliferation  of  requirements  avoided? 


16.  In  what  ways  has  system  software  affected  requirements?  (For  example, 
synchronous  environments  can  restrict  the  flexibility  of  requirements.) 


17.  What  types  of  efficiency  problems  are  major  cost  drivers?  Check  those 
that  apply. 

Restrictions  forced  by  lack  of  computer  time  and 
computer  memory 

System  development  costs 

_  Costs  of  changing  requirements 

_  Lack  of  proper  personnel  training 

_  Other  (explain)  _ 


COMMENTS 

If  you  have  any  comments  on  the  questionnaire  or  related  topics,  please 
use  the  space  below. 


APPENDIX  II 


QUESTIONS 
FOR  THE 

PDSS  WORKING  GROUP 


Questions  for  the  PDSS  Working  Group 


1.  Discuss  differences  between  centralized  PDSS  by  command  and  centralized 
PDSS  by  Battlefield  function. 


2.  In  the  summary,  under  Section  3.5.3: 

a.  How  do  you  view  the  relative  complexities  and  perturbations 
associated  with  Alternative  4  and  Alternative  2? 

b.  Is  it  also  true  that  BFA,  as  well  as  PDSS,  is  a  continuing 
development  process?  (That  is,  whereas  the  developing 
commands  evolve  technology,  the  BFA's  evolve  the  missions). 
From  our  own  experience  we  have  found,  in  fact  that  mission 
requirements  were  much  more  volatile  than  technology  require¬ 
ments.  For  example,  in  the  Apollo  environment,  the  require¬ 
ments  for  mission  phases  like  Boost  and  Entry  changed  con¬ 
tinuously  whereas  guidance,  navigation  and  vehicle  control 
requirements  converged  much  sooner. 


3.  What  other  alternative  implementations  of  the  generalized  software 

support  model  exist  besides  the  5  alternatives  suggested  in  the  executive 
summary?  (For  example.  Alternative  6:  decentralize  by  real  user  in  the 
field).  Alternative  7:  centralize  those  functions  which  are  in  common 
with  respect  to  command  and  centralize  those  functions  which  are  in  common 
with  respect  to  BFA.  Decentralize  those  functions  which  uniquely  use  those 
common  functions. 


4.  What  recommendations  have  been  made  in  the  area  of  front  end  requirements 
definition  which  could  help  to  bridge  the  gap  between  the  user  in  the 
field  (i.e.,  the  real  user)  and  the  system  support  expert.  For  example, 

a.  If  the  software  "code"  were  at  the  higher  level  of  require¬ 
ments  definition,  the  user  could  fix  it  in  real  time  in  his 
own  language. 

b.  If  users  could  speak  the  language  of  the  support  system,  the 
need  for  additional  experts  could  be  alleviated. 

c.  If  system  wide  and  hierarchical  error  detection  and  recovery 
techniques  were  incorporated  into  the  system  for  user  re¬ 
sponse,  the  need  for  adhoc  fixes  would  be  minimized. 

d.  If  all  users'  dialects  could  be  compiled  to  a  common  meeting 
ground,  users  could  be  transferable  from  system  to  system. 


A  II-l 


5. 


What  arguments  would  you  provide  to  demonstrate  the  cost  effectiveness 
of  a  PDSS  plan? 


6.  How  does  a  proponent  in  the  plan  (as  designated  in  Figure  5  of  the  Summary) 
resolve  interface  inconsistencies  with  respect  to  his  own  system  in  the 
case  where  he  has  to  deal  with  more  than  one  support  center? 


7.  What  are  the  differences  between  the  Army  System  in  peacetime  and  the  Army 
System  in  wartime?  See  Summary*  Section  6,  number  2.  The  reason  for  being 
of  the  Army  System  must  consider  this  issue  by  the  very  fact  of  its  existence. 
Thus,  that  which  supported  it  in  non-wartime  would  be  required  to  work  in 
wartime.  This  does  not  preclude,  however,  additional  reconfiguruable  measures 
that  would  be  required  in  wartime.  The  point  is  that  some  systems  are  static* 
and  ready  to  go  in  peacetime  whereas  wartime  is  merely  the  dynamic  operation  of 
those  "static"  systems.  Has  the  working  group  considered  a  preliminary  "what" 
of  this  issue  with  an  example  of  a  "how"? 


8.  See  Summary,  for  example,  Section  4.3.  Could  a  user  training  program 
be  combined  with  the  user  performing  operational  testing  where  off- 
nominal  use  would  provide  stress  testing  that  otherwise  might  not  take 
place  prior  to  battlefield  use?  (In  this  way  independent  verification  and 
validation  might  provide  a  back-up  to  the  nominal  testing  provided  at  the 
development  command.  In  addition,  training  in  use  of  the  system  is  in¬ 
herently  provided  for.) 


9.  Why  does  the  concept  in  Figure  3  (Summary)  best  support  concepts  in  Figure  1 
and  Figure  2  (Summary)? 

a.  What  is  the  difference  between  control,  direct,  monitor, 
approve,  and  manage  (i.e.,  system  manager’s  mission)? 

b.  Does  the  system  manager  report  to  the  interoperability 
configuration  manager?  (Figure  3  of  Executive  Summary.) 

c.  Why  is  the  generic  function  of  the  system  manager  different 
from  that  of  the  application  software  support  manager  (e.g., 
definition  of  requirements)? 

d.  Clarify  the  field  office  versus  system  manager  with  respect 
to  defining  and  resolving  system  problems. 


A 


A  1 1 -2 


m 


10.  Since  most  current  PDSS  systems  have  their  own  system  software  and  com¬ 
puter  types,  and  since  MCF  is  not  presently  available,  how  would  each 
PDSS  center  accommodate  all  of  these  diverse  systems? 


11.  What  is  a  reasonable  plan  for  the  transition  from  the  present  PDSS  efforts 
to  the  new  approach? 
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APPENDIX  Ml 


PRELIMINARIES  OF 


SYSTEM  DEFINITION 


To  Solve  a  problem  in  Physics ,  you  need  to  have  three  basic  units: 


•  Mass 

•  Distance 

•  Time 

All  other  "units"  -  e.g.,  velocity,  acceleration,  momentum,  energy,  etc. 
can  be  expressed  in  terms  of  the  three  basic  units. 

To  solve  a  problem  in  Systems ,  you  also  need  to  have  three  basic  units: 

•  Data  Types 

•  Functions 

•  Control  Structures 

Other  useful  "units"  -  e.g.,  operations,  structures,  etc.  can  be  defined 
in  terms  of  the  three  basic  ones. 

A  System  can  be  represented  as  a  Control  Map 


Each  Node  of  a  control  map  represents  a  Function: 


Function  f 


Function 


All  1-2 


Each  Function  on  a  control  map  involves  Data  Types  playing  two  roles 
Inputs  and  Outputs: 


Function 


Each  level  of  the  control  map  completely  replaces  the  function  at  the 
node  directly  above  it. 


Each  stopping  point  means  a  function  is  reached  whose  behavior  i.e., 
its  input/output  relationship: 


APPENDIX  IV 


FORMAL  DEFINITIONS  USED 


TO  DEFINE  THE  ARMY  LIFE  CYCLE 


TABLE  OF  CONTENTS 


A.  Control  Structures 

B.  Data  Types 


AIV-1 


A  Control  Structure  For  Asynchronous 
Communi eating  Parallel  Processes 


AIV-2 


When  two  functions  coimunicate  asynchronously  an  instance  of  one 
communicates  with  an  instance  of  the  other  after  one  interrupts  the  other. 
In  this  structure  definition,  each  instance  of  the  particular  function 
uses  the  most  recent  information  available  from  its  own  last  instance  and 
the  last  completed  instance  of  the  other  function  in  order  to  produce  its 
own  next  result.  If  an  instance  of  both  functions  are  "ready"  at  the 
same  time,  the  function  of  higher  priority  is  that  one  mentioned  first  in 
the  syntax  statement.  If  there  is  no  contention  for  time,  both  function 
instances  may  run  concurrently.  More  thar.  one  instance  of  one  of  these 
functions  may  occur  before,  or  during,  or  after  an  instantiation  of  the 
other  instance. 


The  interaction,  or  relationship  between  the  two  functions  can  be  seen 
in  the  control  map  definition  in  Figures  A  I V- 1  ,  A  IV-2  and  A  IV-3. 

In  these  figures,  each  subscripted  "x"  refers  to  a  variable  whose  value 
is  of  data  type  State,  and  each  subscripted  "t"  refers  to  a  variable 
whose  value  is  of  type  Time.  The  syntax  for  this  structure  of  asynchronous 
communication  is 


XN’XG  =  N@Pn  O  G@Pg(XN0’XGo,1:) 
Here,  G  and  N  are  functions  of  the  form 


|  Stop? 


where 


xNi+i  =  N(xNi,XGk,tNi); 


-  G(xG,'xNj>tGj> 


and 


=  Ssucc(xN  ) ;  Xq  =  Ssucc(xq  ) 


i+1 


i+l 


Stime(xr  )  <  tM  <  Stime(xr  ) 
\  Ni  “  bk+l 


Stime(xw  )  <  tr  <  Stime(xN  ) 
bi  j+1 


nj 


Stop?(x.,  ,  t.,  ,x«  ,t^  ) 
i  i  j  j 


AIV-3 


is  a  boolean  valued  function  that  defines  the  condition  for  terminating 
the  execution  of  N  and  G, 

tN  is  the  time  that  the  ith  instantiation  of  N 

is  scheduled  to  begin  execution; 


and 

t~  is  the  time  that  the  ith  instantiation  of  G 
Gi 

is  scheduled  to  begin  execution; 
t^  is  calculated  by  function  Pn 
tg  is  calculated  by  function  Pg 

i 

where 


i+1 


=  Pn  ( t 


N . 


GJ 


) 


Likewise, 


t 


G 


i+1 


) 


The  last  instance  of  G  produces  Xg  and  the  last  instance  of  N  produces  x^. 

The  control  map  of  Figure  AIV-1  assumes  that  the  happening  of  two  particular 

States  and  a  Time  indicates  the  execution  of  a  particular  G  or  N.  In 

Figure  AIV-1,  the  time  to  initiate  the  first  instance  of  N  (i.e.,  tw  )  and 

N0 

the  times  to  initiate  the  first  instance  of  G  (i.e.,  tr  )  is  produced  by 

f i .  The  offspring  of  f-j  use  the  Structure  Nextime  (defined  in  Figure 

AIV-4)  "plugging  in"  functions  Pn  and  Pg  to  produce  tw  and  tr  respectively. 

N 0  b0 

If  the  stopping  criteria  is  met  (c.f.  function  Stop?  in  Figure  AIV-1)  the 
initial  input  values  are  assigned  to  the  final  outputs  x^  and  Xg  and  struc- 


AIV-4 


ture  (T)  is  completed.  If  the  stopping  criteria  is  not  met,  either  an 
instance  of  function  N  or  an  instance  of  function  G  will  be  initiated  de¬ 
pending  on  tha  values  of  tM  and  tr  .  The  selection  of  which  particular 

N0  b0 

function  to  initiate  first  is  integrated  at  function  in  Figure  AIV-1. 

The  offspring  of  use  the  structure  First?  (defined  in  Figure  AIV-2) 

"plugging  in"  the  appropriate  functions  in  each  case.  If  tM  <  tr  ,  an 

b0 

instance  of  N  is  initiated  first;  whereas  if  tM  >  tr  ,  an  instance  of 

N0  G0 

G  is  initiated  first..  The  First?  structure  then  defines  the  conditions 
under  which  another  instance  of  the  first  selected  function  is  initiated 
again,  the  conditions  under  which  the  second  function  is  first  initiated 
and  the  conditions  under  which  the  second  function  is  initiated  again. 

When  G  is  initiated  first,  the  ordering  of  the  outputs  and  Xg  is 
specified  by  using  the  Flip  structure  (defined  in  Figure  AIV-3)  in  the 
use  of  the  First?  structure.  Within  structure  (7)  ,  Q  is  initiated 
recursively  via  the  use  of  the  First?  structure.  This  recursion  syn¬ 
chronizes  the  asynchronous  behavior  of  the  instance  of  N  and  the  instances 
of  G  with  respect  to  each  other. 

Since  there  is  a  primitive  operation  on  type  time  to  advance  time, 
but  not  one  to  reverse  time,  one  can  assume  that  in  the  "  (7)  "  struc¬ 
ture  definition,  the  output  times  are  all  greater  than  or  equal  to  the 
input  times.  Thus,  for  example,  the  particular  sequence  of  events  illus¬ 
trated  in  Figure  AIV-5  could  occur. 


CO  JO  IN 


(VvVVlSt0P?<xy  VvV  ■  False 


xN,xG  =  Fi rst?^Pn^sto^? ,G,Pg,idj-^ (XMq  ^Nq  ,XGq  ’  ”Gq^  1 <  t 


G,Pg,Fl  i Pg,Stop? ,N,Pn> id^  4j 


(xQ  ,tG  ,xN  ) 


>  tr 


Syntax :  *N,xG  ’  N@Pn  O  G9Pg(xN’xG  |  |  Stop? 


Fig.  AIV-1  Structure  © 


x. »xD  =  First?(x«  »t*  ,Xn  »tp  } 

A  B/\  Ao  Ao  Bo  Bo 


xA’xBA^1^A0,xB0,tB0,tA^ 
/ COJOIN\ 


Xa  =  ft(xA  *XR  *^A  ^ 

Ro  Bo  Ro 


x,  ,x.  =f,(x.  ,xR  ,tp  ,x.  ,t.  )  t.  =  Newtiroe  (xft  .x.  ,x.  ] 

A  B/^  A0  B0  B0  A1  A1  1  0  B0  M1 


XA,XB  =  QCxA1,xB0’tA1,tB0)|tB^  >  Stime(xft^ 


/or\ 


XA’XB  =  ^VVVVVI^W/a/b^5  =  True 


rVVvVis w(hfy'X\)  - False 


»XA  'XB 
A1  A1  6] 


Xp  =  B(x«  »Xfl  *^R  ^ 

B1  B0  Ao  Bo 


tn  -  ^c(xa  ,XR  >XA  *^A  *x3,  ^ 
A  ^  A0  B0  1  1  1 


B^A^Bj'ltg  >  Stime(xA^ 
<  Stime(xA  ) 


tn  -  NewtimeDK (x„  ,x„  ,x„  ) 


Pb  xBq’xAq’  b-j  '  |Stime(xB^ )  <  $time(xA^) 


tc  =  NewtimeDh(xa  ,x.  ,xR  ) 


Bo,xA1  ’  |Stime(xB^)  >  Stime(xA^) 


Syntax: 


XA,XB  =  First? 


A,Pa,Q,Stop?,B,Pb,S(xA0’tA0’x50,tS0) 


Fig.  AIV-2  Structure  First? 


tl  =  Sl{xA,XB’V 


Syntax:  t,  =  NextTime-  (x.,xD,t  ) 

I  A  o  0 


Syntax:  t,  =  Newtimec(x.  ,xD,xA  ) 

1  F  Ao  B  A1 


Fig.  AIV-4  Structures  Nextime  and  Newtime 


time 


Fig.  AIV-5  Example  of  G  &  N  Communicating  in  Parallel  and 

Asynchronously 


A I V  - 1 0 


The  Failure  Structure 


AIV-11 


The  failure  structure,  the  definition  of  which  follows, 
provides  for  the  ability  to  “recover”  from  a  “detected'' 
error.  The  definition  uses  the  cojoin,  coor,  join,  and  each 
structures. 

Structure:  y  **  Failure(x): 

where  t.r,  g,  y,  ,vl0j)  are  of  some  type, 

a  is  an  Ordered  Set  (of  Naturals): 

Failure:  y  -  /,(x  g)  cojoin  g  -  E(x): 

ft-  >'  -  Clone, (g)  coor  y  =  /2(x)  ; 

ft-  y  =  join  x loj  =  id,0,(x); 

syntax: y  =  £(x)  failure  y  -  £(xul); 

end  Failure: 
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DATA  TYPE:  TIME; 


PRIMITIVE  OPERATIONS: 
time^  =  Advance (time1,time2) ; 
boolean  =  Notaf ter ( time1 , time2) ; 
boolean  =  Equal  (time^time^  ; 

AXIOMS : 

WHERE  t,t1,t2,t3  ARE  TIMES; 

WHERE  Notime  IS  A  CONSTANT  TIME; 


1. 

Equal (t,t)  =  True; 

2. 

Equal (t1,t2)  =  Equal (t2 

/t1)  ; 

3. 

Entails (Equal (t^ , t2)  &  Equal (t2 , t^) ,  Equal (t^,t3) )  True; 

4. 

Notafter (t,t)  =  True; 

5. 

Entails (Notafter (t^, t2) 

&  Notafter (t2,t3) ,  Notafter (t1, t3) ) 

True ; 

6. 

Entails (Notafter (t1,t2) 

&  Notafter (t2,t1) ,  Equal , t2) )  « 

True ; 

7. 

Notafter  (trt2)  !  Notaf  ter  Ct2  ,  t^  =  True; 

8. 

Advance (t ,No time)  =  t; 

9. 

Advance (tx,t2)  =  Advance Ct2 , t^  ; 

10. 

Advance (t^ /Advance (t2  / 1 

3))  =  Advance (Advance (t1,t2) ,t3) ; 

11. 

Notafter (Advance ( t1 ,  t2) 

,  t^  =  Notaf  ter  (t2, Notime)  ; 

END 

TIME? 

Data  Type:  Stale  (of  Tl, 
primitive  operations: 
time  «  Stimel state); 
r  =  Comespondentistate). 
state,  =  Ssucc( state,): 

Boolean  =  Sequalslstate,.  state.l; 
axioms: 

where  ( s„  J,)  are  States  (of  T), 
time  is  a  Time. 

/  is  a  7; 

Precedes'VStimeU,).  Stime(Ssucc(s,))  =  True; 
Equals(Coirespondent(j,l.  ConespondenKr.)) 

*  False  C  Stimeu,)  *  Stimel s.)  =  True; 
SequalsU,.  j,)  =  Equals) Stimel j,i,  Stimeu,)) 

And  Equals(Correspondent(  J,).  CorrespondentU,)); 
end  State  (of  T); 
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MONEY 


DATA  TYPE :  MONEY; 


Primitive  Operations : 


BOOLEAN  =  floRETHAN  (MONEY},  M0NEY2); 
/*MONEY  IS  ORDERED*/ 


money3  =  Total  (money-),  money2); 


/*MONEY  ADDS*/ 


From  "Some  Characterizations  of  Resources,"  S.  Cushing  (DCPA  Memo 
in  preparation) . 
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AXIOMS 


tk 


COMMODITY 


person2  =  Seller  (commodity,  time,  person1 
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/* MONEY  IS  THE  VALUE  OF  A  COMMODITY  AT  A  TIME*/ 
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APPENDIX  V 


THREE  PRIMITIVE 


CONTROL  STRUCTURES 


Figure  A2.  Class  partition. 


The  primitive  control  structures  form  the  basis  for  de¬ 
fining  other  control  structures  in  axes.  The  use  of  axes 
syntax  and  associated  rules  for  the  primitive  control  struc¬ 
tures  follow: 

For  composition,  if  y  =  /0U), 

U  .'■=/«(£>  join  g  =  ft(x): 

(See  Figure  At.) 

1 .  One  and  only  one  offspring  (specifically,  /,  in  this  exam¬ 
ple)  receives  access  rights  to  the  input  data  x  from /*. 

2.  One  and  only  one  offspring  (specifically,/,  in  this  exam¬ 
ple)  has  access  rights  to  deliver  the  output  data  y  for  /„. 

3.  All  other  input  and  output  data  that  will  be  produced  by 
offspring,  controlled  by  /„,  will  reside  in  local  variables 
(specifically,  "g"  in  this  example).  Local  variable  "g" 
provides  communication  between  the  offspring/,  and/,. 


y  *  f0M 


Figure  Al.  Composition. 


4.  Even-  offspring  is  specified  to  be  invoked  once  and  only 
once  in  each  process  of  performing  its  parent  s  corre¬ 
sponding  function. 

5.  Every  local  variable  must  exist  both  as  an  input  variable 
for  one  and  only  one  function  and  as  an  output  variable 
for  one  and  only  one  different  function  on  the  same 
level. 

For  Class  partition,  if  (>•„  y.)  =  /,/ jr,.  jj), 

fa-  v,  =  /,U,) include  y,  =  /, U,); 

(See  Figure  A2.) 

1 .  All  offspring  of  /0  are  granted  permission  to  receive 
input  values  taken  from  a  partitioned  variable  in  the  set 
of  the  parent's  corresponding  function  domain  varia¬ 
bles.  such  that  each  offspring's  set  of  input  variables 
collectively  represents  the  parent's  corresponding  func¬ 
tion  input  variables. 

2.  All  offspring  of  f„  are  granted  permission  to  produce 
output  values  for  a  partitioned  variable  in  the  set  of  the 
parent's  corresponding  function  range  variables,  such 
that  the  sets  of  each  offspring's  output  variables  collec¬ 
tively  represent  the  parent's  corresponding  function 
variables. 

3.  Each  offspring  is  specified  to  be  invoked  per  input  value 
received  for  each  process  of  performing  its  parent's 
corresponding  function. 

4.  There  is  no  communication  between  offspring. 

For  set  partition,  if  y  -  /rf. r), 

fo-y=M x)  or  v  -/,(*)!  ; 

**>**«>  i* 

(See  Figure  A3.) 


J'l'W  y2  “  W 


Figure  A3.  Set  partition. 


1.  Every  offspring  of  the  parent  at  /„  Is  granted  permission 
to  produce  output  values  of  “v." 

2.  All  offspring  of  the  parent  at  /„  are  granted  permission 
to  receive  input  values  from  the  variable  “x." 

3.  Only  one  offspring  is  specified  to  be  invoked  per  input 
value  received  for  each  process  of  performing  its  par¬ 
ent's  corresponding  function. 

4.  The  values  represented  by  the  input  variables  of  an 
offspring's  function  comprise  a  proper  subset  of  the 
domain  of  the  function  of  the  parent. 

5.  There  is  no  communication  between  offspring. 

In  the  above  definitions  x,  y,  y ,,  y2,  x„  x.  are  ordered  sets 
of  variables:  /<*  /„  are  functions;  property  is  of  type 
Property  (of  T)  [19];  and  Pnot  is  a  primitive  operation  on 
type  property  whose  result  is  a  property  exclusive  of  its 
input  argument. 
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Figure  A 2.  Class  partition. 


The  primitive  control  structures  form  the  basis  for  de¬ 
fining  other  control  structures  in  axes.  The  use  of  axes 
syntax  and  associated  rules  for  the  primitive  control  struc¬ 
tures  follow: 

(  For  composition,  if  v  «  /„(jr), 

/«:.'•  -/i(g)  join  g  -  f,U); 

(See  Figure  Al.) 

1 .  One  and  only  one  offspring  (specifically,  /,  in  this  exam¬ 
ple)  receives  access  rights  to  the  input  data  x  from  /». 

2.  One  and  only  one  offspring  ( specifically,  /,  in  this  exam- 

♦  pie)  has  access  rights  to  deliver  the  output  data  y  for  /„. 

3.  All  other  input  and  output  data  that  will  be  produced  by 
offspring,  controlled  by /0:  will  reside  in  local  variables 
(specifically,  “ g "  in  this  example).  Local  variable  "g" 
provides  communication  between  the  offspring/j  and 


.  y  *  f0(x) 


Figure  Al.  Composition. 


4.  Even-  offspring  is  specified  to  be  invoked  once  and  only 
once  in  each  process  of  performing  its  parent's  corre¬ 
sponding  function. 

5.  Even’  local  variable  must  exist  both  as  an  input  variable 
for  one  and  only  one  function  and  as  an  output  variable 
for  one  and  only  one  different  function  on  the  same 

*  level. 

For  Class  partition,  if  (y ,,  y.l  =  x  J, 

U  .'I  include  y,  ‘f^,xt): 

(See  Figure  A2.) 

!.  All  offspring  of  /0  are  granted  permission  to  receive 

♦  input  values  taken  from  a  partitioned  variable  in  the  set 
of  the  parent's  corresponding  function  domain  varia¬ 
bles.  such  that  each  offspring's  set  of  input  variables 
collectively  represents  the  parent's  corresponding  func¬ 
tion  input  variables. 

2.  All  offspring  of  /„  are  granted  permission  to  produce 
output  values  for  a  partitioned  variable  in  the  set  of  the 
parent's  corresponding  function  range  variables,  such 
that  the  sets  of  each  offspring's  output  variables  coilec- 
tiveh  represent  the  parent's  corresponding  function 
variables. 

3.  Each  offspring  is  specified  to  be  invoked  per  input  value 
received  for  each  process  of  performing  its  parent's 
corresponding  function. 

4.  There  is  no  communication  between  offspring. 

For  set  partition,  if  y  mfjxi. 

/►'  v  “fix'  ory-/,(jt), 

*******  'P*Wi*f***rV  i* 

|  (See  Figure  A3.) 


1.  Every  offspring  of  the  parent  at  /,  is  granted  permission 
to  produce  output  values  of  “v." 

2.  All  offspring  of  the  parent  at  /„  are  granted  permission 
to  receive  input  values  from  the  variable  "x." 

3.  Only  one  offspring  is  specified  to  be  invoked  per  input 
value  received  for  each  process  of  performing  its  par¬ 
ent's  corresponding  function. 

4.  The  values  represented  by  the  input  variables  of  an 
offspring's  function  comprise  a  proper  subset  of  the 
domain  of  the  function  of  the  parent. 

5.  There  is  no  communication  between  offspring. 

In  the  above  definitions  x,  y,  v,,  y2,  x„  x-  are  ordered  sets 
of  variables;  /„  /-  are  functions:  property  is  of  type 
Property  (of  T)  [19];  and  Pnot  is  a  primitive  operation  on 
type  property  whose  result  is  a  property  exclusive  of  its 
input  argument. 
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